2023-09-30T01:45:08.527 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T01:45:08.527 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --description rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135 -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/orig.config.yaml 2023-09-30T01:45:08.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:45:08.666 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} email: yweinste@redhat.com first_in_suite: false job_id: '7407135' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug cephsqlite: 20 debug client: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - POOL_APP_NOT_ENABLED - do not have an application enabled sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_system_packages: deb: - sqlite3 rpm: - sqlite-devel - ceph: null - exec: client.0: - ceph osd pool create cephsqlite - ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring - exec: client.0: - ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr - workunit: clients: client.0: - rados/test_libcephsqlite.sh cephsqlite env: CEPH_ARGS: --id libcephsqlite --no-log-to-stderr teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T01:45:08.667 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T01:45:08.668 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T01:45:08.668 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T01:45:08.670 INFO:teuthology.task.internal:Checking packages... 2023-09-30T01:45:08.692 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T01:45:08.692 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T01:45:08.692 INFO:teuthology.packaging:ref: None 2023-09-30T01:45:08.692 INFO:teuthology.packaging:tag: None 2023-09-30T01:45:08.692 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:45:08.692 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:45:08.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:45:08.907 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:45:08.907 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T01:45:08.913 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T01:45:08.913 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T01:45:08.919 INFO:teuthology.task.internal:Saving configuration 2023-09-30T01:45:08.934 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T01:45:08.941 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:45:08.966 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:38:57.558491', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T01:45:08.985 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:38:57.557219', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T01:45:08.986 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T01:45:08.992 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T01:45:08.992 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T01:45:08.992 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T01:45:09.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8f82371040>, signals=[15]) 2023-09-30T01:45:09.067 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T01:45:09.073 INFO:teuthology.task.internal:Opening connections... 2023-09-30T01:45:09.073 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:45:09.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:45:09.122 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:45:09.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:45:09.168 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T01:45:09.175 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2023-09-30T01:45:09.203 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2023-09-30T01:45:09.203 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2023-09-30T01:45:09.250 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Ubuntu" 2023-09-30T01:45:09.250 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:ID=ubuntu 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE=debian 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="20.04" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T01:45:09.251 INFO:teuthology.orchestra.run.smithi151.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T01:45:09.252 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_CODENAME=focal 2023-09-30T01:45:09.252 INFO:teuthology.orchestra.run.smithi151.stdout:UBUNTU_CODENAME=focal 2023-09-30T01:45:09.252 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-09-30T01:45:09.276 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-09-30T01:45:09.284 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-09-30T01:45:09.284 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="20.04" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T01:45:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T01:45:09.334 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=focal 2023-09-30T01:45:09.334 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=focal 2023-09-30T01:45:09.334 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-09-30T01:45:09.359 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T01:45:09.369 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T01:45:09.373 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T01:45:09.373 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:45:09.376 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:45:09.381 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T01:45:09.388 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T01:45:09.388 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:45:09.422 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:45:09.438 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T01:45:09.499 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T01:45:09.514 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T01:45:09.514 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T01:45:09.514 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:45:09.515 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:45:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:45:09.520 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2023-09-30T01:45:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:45:09.521 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-09-30T01:45:09.527 INFO:teuthology.orchestra.run.smithi151.stdout:5.4.0-124-generic 2023-09-30T01:45:09.528 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.4.0-124-generic 2023-09-30T01:45:09.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get clean 2023-09-30T01:45:09.529 INFO:teuthology.orchestra.run.smithi172.stdout:5.4.0-124-generic 2023-09-30T01:45:09.529 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.4.0-124-generic 2023-09-30T01:45:09.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2023-09-30T01:45:09.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-09-30T01:45:09.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2023-09-30T01:45:09.841 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:45:09.841 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T01:45:09.841 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T01:45:09.869 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:45:09.870 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T01:45:09.903 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T01:45:09.944 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T01:45:09.965 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T01:45:10.228 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T01:45:10.268 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T01:45:10.378 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T01:45:10.379 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T01:45:10.391 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T01:45:10.393 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T01:45:10.394 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T01:45:10.394 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T01:45:10.407 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T01:45:10.409 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T01:45:10.412 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:45:10.412 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T01:45:10.416 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T01:45:10.420 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T01:45:10.426 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T01:45:10.426 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T01:45:10.426 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T01:45:10.426 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T01:45:10.429 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T01:45:10.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T01:45:10.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T01:45:10.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T01:45:10.454 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T01:45:10.454 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T01:45:10.457 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:45:10.457 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T01:45:10.466 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T01:45:10.480 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T01:45:10.480 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T01:45:10.486 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T01:45:10.486 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T01:45:10.494 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T01:45:10.494 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T01:45:10.494 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T01:45:10.494 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T01:45:10.494 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T01:45:10.495 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T01:45:10.495 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T01:45:10.495 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T01:45:10.512 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T01:45:10.512 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T01:45:10.512 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T01:45:10.513 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T01:45:10.517 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T01:45:10.517 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T01:45:10.523 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:45:10.523 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T01:45:10.532 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T01:45:10.538 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T01:45:10.539 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T01:45:10.540 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T01:45:10.540 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T01:45:10.540 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T01:45:10.540 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T01:45:10.553 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T01:45:10.554 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T01:45:10.554 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T01:45:10.570 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T01:45:10.571 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T01:45:10.571 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T01:45:10.581 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T01:45:10.906 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T01:45:11.093 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T01:45:11.107 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T01:45:11.109 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T01:45:11.176 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T01:45:11.176 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T01:45:11.182 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:45:11.182 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T01:45:11.201 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T01:45:11.393 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T01:45:11.809 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T01:45:11.815 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T01:45:11.817 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T01:45:11.826 INFO:teuthology.orchestra.run.smithi151.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T01:45:11.827 INFO:teuthology.orchestra.run.smithi151.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T01:45:14.224 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 17.4 MB in 3s (6,745 kB/s) 2023-09-30T01:45:14.425 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 17.4 MB in 3s (6,783 kB/s) 2023-09-30T01:45:15.615 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:45:15.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:45:15.705 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:45:15.904 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-09-30T01:45:15.904 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-09-30T01:45:15.933 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:45:15.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:45:16.041 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:45:16.084 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-09-30T01:45:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:45:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T01:45:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T01:45:16.085 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-09-30T01:45:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T01:45:16.085 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2023-09-30T01:45:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2023-09-30T01:45:16.141 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-09-30T01:45:16.142 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:45:16.142 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T01:45:16.142 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T01:45:16.143 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2023-09-30T01:45:16.143 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T01:45:16.242 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2023-09-30T01:45:16.243 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2023-09-30T01:45:16.326 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:45:16.326 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 77.1 MB of archives. 2023-09-30T01:45:16.327 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T01:45:16.327 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T01:45:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2023-09-30T01:45:16.433 INFO:teuthology.orchestra.run.smithi151.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:45:16.433 INFO:teuthology.orchestra.run.smithi151.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T01:45:16.433 INFO:teuthology.orchestra.run.smithi151.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T01:45:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2023-09-30T01:45:16.434 INFO:teuthology.orchestra.run.smithi151.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T01:45:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:Recommended packages: 2023-09-30T01:45:16.434 INFO:teuthology.orchestra.run.smithi151.stdout: thermald 2023-09-30T01:45:16.497 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2023-09-30T01:45:16.497 INFO:teuthology.orchestra.run.smithi151.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:45:16.497 INFO:teuthology.orchestra.run.smithi151.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T01:45:16.497 INFO:teuthology.orchestra.run.smithi151.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T01:45:16.499 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2023-09-30T01:45:16.499 INFO:teuthology.orchestra.run.smithi151.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T01:45:16.552 INFO:teuthology.orchestra.run.smithi151.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:45:16.552 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 77.1 MB of archives. 2023-09-30T01:45:16.552 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T01:45:16.552 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T01:45:16.851 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T01:45:16.945 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T01:45:17.458 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T01:45:17.461 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T01:45:17.461 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T01:45:17.462 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T01:45:17.552 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T01:45:17.562 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T01:45:17.776 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T01:45:18.289 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 77.1 MB in 1s (72.9 MB/s) 2023-09-30T01:45:18.498 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T01:45:19.029 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T01:45:19.029 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T01:45:19.029 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T01:45:19.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T01:45:19.347 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T01:45:20.017 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 77.1 MB in 3s (24.1 MB/s) 2023-09-30T01:45:20.206 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T01:45:20.275 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T01:45:20.280 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:20.463 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:21.965 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T01:45:21.970 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:22.070 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:22.526 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T01:45:22.539 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:22.626 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:23.772 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T01:45:23.786 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:23.863 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:23.971 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T01:45:23.984 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:24.080 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:25.132 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T01:45:25.145 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:25.181 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:28.826 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:28.975 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:29.325 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:29.464 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:29.810 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T01:45:29.827 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T01:45:29.868 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:45:29.912 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:30.038 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:30.370 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:30.508 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:30.861 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T01:45:30.875 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T01:45:30.919 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:45:36.050 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T01:45:36.073 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:36.116 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:36.319 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T01:45:36.339 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:45:36.376 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:38.981 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:39.129 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:39.491 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:40.000 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:45:40.137 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:45:40.462 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:40.780 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:45:40.915 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:41.689 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:45:41.819 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:42.394 INFO:teuthology.orchestra.run.smithi151.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:45:42.395 INFO:teuthology.orchestra.run.smithi151.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T01:45:42.970 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:45:42.971 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T01:45:43.104 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:43.109 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:43.230 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:43.235 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:44.437 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T01:45:44.589 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T01:45:44.740 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T01:45:44.875 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:44.890 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T01:45:44.950 INFO:teuthology.orchestra.run.smithi151.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T01:45:44.950 INFO:teuthology.orchestra.run.smithi151.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:45:45.025 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T01:45:45.167 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T01:45:45.294 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:45:45.369 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T01:45:45.369 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:46:07.047 INFO:teuthology.orchestra.run.smithi151.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T01:46:07.047 INFO:teuthology.orchestra.run.smithi151.stdout:Sourcing file `/etc/default/grub' 2023-09-30T01:46:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:46:07.078 INFO:teuthology.orchestra.run.smithi151.stdout:Generating grub configuration file ... 2023-09-30T01:46:07.421 INFO:teuthology.orchestra.run.smithi151.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:46:07.445 INFO:teuthology.orchestra.run.smithi151.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:46:07.447 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T01:46:07.447 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2023-09-30T01:46:07.469 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:46:07.489 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2023-09-30T01:46:07.827 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:46:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:46:07.885 INFO:teuthology.orchestra.run.smithi151.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:46:07.910 INFO:teuthology.orchestra.run.smithi151.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:46:08.252 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:46:08.263 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:46:09.785 INFO:teuthology.orchestra.run.smithi151.stdout:done 2023-09-30T01:46:10.022 INFO:teuthology.orchestra.run.smithi172.stdout:done 2023-09-30T01:46:10.278 DEBUG:teuthology.orchestra.run.smithi151:> dpkg -s linux-image-generic 2023-09-30T01:46:10.327 INFO:teuthology.orchestra.run.smithi151.stdout:Package: linux-image-generic 2023-09-30T01:46:10.327 INFO:teuthology.orchestra.run.smithi151.stdout:Status: install ok installed 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Priority: optional 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Section: kernel 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Installed-Size: 20 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: amd64 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Source: linux-meta 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Version: 5.4.0.163.160 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:46:10.328 INFO:teuthology.orchestra.run.smithi151.stdout:Recommends: thermald 2023-09-30T01:46:10.329 INFO:teuthology.orchestra.run.smithi151.stdout:Description: Generic Linux kernel image 2023-09-30T01:46:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:46:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: available. 2023-09-30T01:46:10.329 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T01:46:10.329 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:46:10.329 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:46:10.329 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T01:46:10.329 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:46:10.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get clean 2023-09-30T01:46:10.391 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2023-09-30T01:46:10.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.4.0.163.160 2023-09-30T01:46:10.432 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:46:10.433 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:46:10.433 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2023-09-30T01:46:10.433 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2023-09-30T01:46:10.433 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:46:10.433 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2023-09-30T01:46:10.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T01:46:10.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:46:10.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:46:10.433 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T01:46:10.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:46:10.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2023-09-30T01:46:10.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-09-30T01:46:10.557 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:46:10.558 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:46:10.575 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:46:10.595 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:46:10.653 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:46:10.653 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:46:10.664 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:46:10.684 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:46:12.301 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:46:12.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:46:12.394 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:46:12.394 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:46:12.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:46:12.488 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:46:12.574 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2023-09-30T01:46:12.574 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2023-09-30T01:46:12.666 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-09-30T01:46:12.667 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-09-30T01:46:12.779 INFO:teuthology.orchestra.run.smithi151.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T01:46:12.779 INFO:teuthology.orchestra.run.smithi151.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:46:12.781 DEBUG:teuthology.orchestra.run.smithi151:> dpkg -s linux-image-generic 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Package: linux-image-generic 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Status: install ok installed 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Priority: optional 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Section: kernel 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Installed-Size: 20 2023-09-30T01:46:12.804 INFO:teuthology.orchestra.run.smithi151.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: amd64 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Source: linux-meta 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Version: 5.4.0.163.160 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Recommends: thermald 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout:Description: Generic Linux kernel image 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:46:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: available. 2023-09-30T01:46:12.806 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-09-30T01:46:12.810 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.oQR5S507aI 2023-09-30T01:46:12.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oQR5S507aI 2023-09-30T01:46:12.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.oQR5S507aI 2023-09-30T01:46:12.878 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T01:46:12.878 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:46:12.879 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2023-09-30T01:46:12.899 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2023-09-30T01:46:12.899 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2023-09-30T01:46:12.899 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.4.0.163.160 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:46:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2023-09-30T01:46:12.901 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2023-09-30T01:46:12.901 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:46:12.901 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2023-09-30T01:46:12.901 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-09-30T01:46:12.907 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.b6iVXjyVOs 2023-09-30T01:46:12.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.b6iVXjyVOs 2023-09-30T01:46:12.910 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.oQR5S507aI is 9KB 2023-09-30T01:46:12.923 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.oQR5S507aI 2023-09-30T01:46:12.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T01:46:12.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.b6iVXjyVOs 2023-09-30T01:46:12.989 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:46:12.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T01:46:12.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T01:46:13.010 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.b6iVXjyVOs is 9KB 2023-09-30T01:46:13.025 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.b6iVXjyVOs 2023-09-30T01:46:13.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T01:46:13.032 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T01:46:13.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo update-grub 2023-09-30T01:46:13.089 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:46:13.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T01:46:13.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T01:46:13.129 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T01:46:13.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2023-09-30T01:46:14.363 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2023-09-30T01:46:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:46:14.386 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2023-09-30T01:46:14.544 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub' 2023-09-30T01:46:14.550 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:46:14.560 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2023-09-30T01:46:14.634 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:46:14.646 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:46:14.863 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:46:14.875 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:46:15.004 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:46:15.015 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:46:15.296 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:46:15.308 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:46:15.783 INFO:teuthology.orchestra.run.smithi172.stderr:done 2023-09-30T01:46:15.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2023-09-30T01:46:16.048 INFO:teuthology.orchestra.run.smithi151.stderr:done 2023-09-30T01:46:16.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2023-09-30T01:46:45.814 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T01:46:45.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:46:45.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:46:46.053 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T01:46:46.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:46:46.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:47:04.186 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-09-30T01:47:04.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-09-30T01:47:13.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:47:13.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:47:13.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:47:13.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:47:16.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-09-30T01:47:16.506 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-09-30T01:47:28.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:47:28.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:47:28.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:47:28.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:47:46.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-09-30T01:48:01.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:48:01.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:48:02.334 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-09-30T01:48:03.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:48:03.530 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-163-generic"... 2023-09-30T01:48:03.531 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2023-09-30T01:48:03.537 INFO:teuthology.orchestra.run.smithi151.stdout:5.4.0-163-generic 2023-09-30T01:48:03.537 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T01:48:03.537 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:48:03.537 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T01:48:04.539 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T01:48:04.539 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:48:04.654 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2023-09-30T01:48:04.682 DEBUG:teuthology.parallel:result is None 2023-09-30T01:48:28.268 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T01:48:43.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:48:43.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:48:43.479 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-09-30T01:48:44.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:48:44.022 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-163-generic"... 2023-09-30T01:48:44.023 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-09-30T01:48:44.029 INFO:teuthology.orchestra.run.smithi172.stdout:5.4.0-163-generic 2023-09-30T01:48:44.030 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T01:48:44.030 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:48:44.030 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T01:48:45.031 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T01:48:45.031 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:48:45.143 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-09-30T01:48:45.163 DEBUG:teuthology.parallel:result is None 2023-09-30T01:48:45.164 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T01:48:45.171 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T01:48:45.172 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:48:45.175 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:48:45.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T01:48:45.235 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T01:48:45.249 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T01:48:45.249 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:48:45.252 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:48:45.302 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T01:48:45.308 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T01:48:45.309 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:48:45.311 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:48:45.329 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:48:45.329 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:48:45.340 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:48:45.340 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:48:45.342 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T01:48:45.364 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T01:48:45.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:48:45.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:48:45.384 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T01:48:45.392 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T01:48:45.392 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:48:45.426 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:48:45.432 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:48:45.440 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:48:45.447 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:48:45.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:48:45.523 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:48:45.530 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:48:45.539 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:48:45.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:48:45.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2023-09-30T01:48:45.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-09-30T01:48:45.620 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T01:48:45.627 INFO:teuthology.task.internal:Starting timer... 2023-09-30T01:48:45.628 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T01:48:45.641 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T01:48:45.650 DEBUG:teuthology.task.selinux:Excluding smithi151: OS 'ubuntu' does not support SELinux 2023-09-30T01:48:45.650 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2023-09-30T01:48:45.650 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T01:48:45.650 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-30T01:48:45.650 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T01:48:45.650 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T01:48:45.660 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T01:48:45.660 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T01:48:45.678 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T01:48:45.692 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T01:54:35.443 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-09-30T01:54:35.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:54:35.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:54:35.494 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-09-30T01:54:35.612 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-09-30T01:54:35.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:54:35.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:54:35.660 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-09-30T01:54:35.784 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-09-30T01:54:35.784 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T01:54:35.797 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T01:54:35.797 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:54:35.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:54:35.800 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:54:35.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:54:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T01:54:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Command line: ntpd -gq 2023-09-30T01:54:35.834 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: proto: precision = 0.117 usec (-23) 2023-09-30T01:54:35.834 INFO:teuthology.orchestra.run.smithi151.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:54:35.835 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:54:35.835 INFO:teuthology.orchestra.run.smithi151.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen normally on 3 enp3s0f1 172.21.15.151:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen normally on 4 lo [::1]:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f63%5]:123 2023-09-30T01:54:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:35 ntpd[22672]: Listening on routing socket on fd #22 for interface updates 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Command line: ntpd -gq 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: proto: precision = 0.088 usec (-23) 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen normally on 4 lo [::1]:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:35 ntpd[22741]: Listening on routing socket on fd #22 for interface updates 2023-09-30T01:54:36.833 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.883 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:36.884 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:36 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:36.885 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:36.886 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:36.886 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:36.886 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:36.886 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:36 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:38.834 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:38.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:38 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:38.861 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.861 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:38.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:38.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:38 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:40 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:40.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:40 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.835 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:42.836 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.837 INFO:teuthology.orchestra.run.smithi151.stdout:30 Sep 01:54:42 ntpd[22672]: ntpd: time slew +0.003132 s 2023-09-30T01:54:42.837 INFO:teuthology.orchestra.run.smithi151.stdout:ntpd: time slew +0.003132s 2023-09-30T01:54:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T01:54:42.837 INFO:teuthology.orchestra.run.smithi151.stderr:30 Sep 01:54:42 ntpd[22672]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T01:54:42.861 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:54:42.862 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:54:42.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:54:42.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:54:42.863 INFO:teuthology.orchestra.run.smithi172.stdout:30 Sep 01:54:42 ntpd[22741]: ntpd: time slew -0.000406 s 2023-09-30T01:54:42.863 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.000406s 2023-09-30T01:54:42.863 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T01:54:42.864 INFO:teuthology.orchestra.run.smithi172.stderr:30 Sep 01:54:42 ntpd[22741]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T01:54:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:54:42.908 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2023-09-30T01:54:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.931 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:54:42.932 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2023-09-30T01:54:42.932 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.932 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.932 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.932 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:54:42.933 INFO:teuthology.run_tasks:Running task install... 2023-09-30T01:54:42.942 DEBUG:teuthology.task.install:project ceph 2023-09-30T01:54:42.942 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T01:54:42.943 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T01:54:42.943 INFO:teuthology.task.install:Using flavor: default 2023-09-30T01:54:42.949 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T01:54:42.949 INFO:teuthology.task.install:extra packages: [] 2023-09-30T01:54:42.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-key list | grep Ceph 2023-09-30T01:54:42.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2023-09-30T01:54:43.048 INFO:teuthology.orchestra.run.smithi151.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T01:54:43.050 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T01:54:43.106 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T01:54:43.106 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T01:54:43.107 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T01:54:43.107 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T01:54:43.108 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T01:54:43.108 INFO:teuthology.task.install.deb:Installing system (non-project) packages: sqlite3 on remote deb x86_64 2023-09-30T01:54:43.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:54:43.112 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T01:54:43.113 INFO:teuthology.task.install.deb:Installing system (non-project) packages: sqlite3 on remote deb x86_64 2023-09-30T01:54:43.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:54:43.253 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T01:54:43.253 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:54:43.325 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T01:54:43.325 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:54:43.377 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:54:43.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T01:54:43.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2023-09-30T01:54:43.497 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:54:43.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T01:54:43.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2023-09-30T01:54:43.582 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:54:43.590 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:54:43.610 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:54:43.636 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T01:54:43.657 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T01:54:43.680 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T01:54:43.697 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:54:43.702 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T01:54:43.711 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:54:43.742 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T01:54:43.821 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:54:43.830 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T01:54:43.870 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T01:54:43.903 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:54:43.910 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T01:54:43.946 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T01:54:43.984 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:54:44.006 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T01:54:44.222 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 28.4 kB in 1s (46.1 kB/s) 2023-09-30T01:54:44.350 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 28.4 kB in 1s (45.3 kB/s) 2023-09-30T01:54:45.252 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:54:45.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:54:45.346 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:54:45.369 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:54:45.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:54:45.462 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:54:45.546 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-09-30T01:54:45.547 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-09-30T01:54:45.641 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2023-09-30T01:54:45.641 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2023-09-30T01:54:45.727 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T01:54:45.729 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T01:54:45.730 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T01:54:45.730 INFO:teuthology.orchestra.run.smithi172.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T01:54:45.730 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T01:54:45.730 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T01:54:45.731 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: socat xmlstarlet 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T01:54:45.732 INFO:teuthology.orchestra.run.smithi172.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2023-09-30T01:54:45.733 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T01:54:45.814 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T01:54:45.815 INFO:teuthology.orchestra.run.smithi151.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T01:54:45.815 INFO:teuthology.orchestra.run.smithi151.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T01:54:45.815 INFO:teuthology.orchestra.run.smithi151.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T01:54:45.815 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T01:54:45.815 INFO:teuthology.orchestra.run.smithi151.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T01:54:45.816 INFO:teuthology.orchestra.run.smithi151.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: socat xmlstarlet 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T01:54:45.817 INFO:teuthology.orchestra.run.smithi151.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:Recommended packages: 2023-09-30T01:54:45.818 INFO:teuthology.orchestra.run.smithi151.stdout: btrfs-tools 2023-09-30T01:54:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-09-30T01:54:45.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T01:54:45.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T01:54:45.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T01:54:45.844 INFO:teuthology.orchestra.run.smithi172.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T01:54:45.845 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T01:54:45.846 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T01:54:45.846 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T01:54:45.846 INFO:teuthology.orchestra.run.smithi172.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T01:54:45.846 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2023-09-30T01:54:45.846 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 librbd1 2023-09-30T01:54:45.912 INFO:teuthology.orchestra.run.smithi172.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T01:54:45.912 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 209 MB of archives. 2023-09-30T01:54:45.912 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T01:54:45.913 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T01:54:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2023-09-30T01:54:45.933 INFO:teuthology.orchestra.run.smithi151.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T01:54:45.933 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T01:54:45.933 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T01:54:45.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T01:54:45.936 INFO:teuthology.orchestra.run.smithi151.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T01:54:45.937 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2023-09-30T01:54:45.937 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 librbd1 2023-09-30T01:54:45.993 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T01:54:46.007 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T01:54:46.009 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T01:54:46.019 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T01:54:46.027 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T01:54:46.062 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T01:54:46.103 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T01:54:46.106 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T01:54:46.112 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T01:54:46.114 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T01:54:46.114 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T01:54:46.116 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T01:54:46.116 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T01:54:46.117 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T01:54:46.118 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T01:54:46.124 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T01:54:46.130 INFO:teuthology.orchestra.run.smithi151.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T01:54:46.130 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 209 MB of archives. 2023-09-30T01:54:46.130 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T01:54:46.130 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T01:54:46.139 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T01:54:46.139 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T01:54:46.139 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T01:54:46.141 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T01:54:46.141 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T01:54:46.142 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T01:54:46.143 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T01:54:46.143 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T01:54:46.145 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T01:54:46.163 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T01:54:46.163 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T01:54:46.170 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T01:54:46.170 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T01:54:46.171 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T01:54:46.171 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T01:54:46.172 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T01:54:46.182 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T01:54:46.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T01:54:46.185 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T01:54:46.186 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T01:54:46.202 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T01:54:46.204 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T01:54:46.206 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T01:54:46.206 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T01:54:46.207 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T01:54:46.207 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T01:54:46.305 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T01:54:46.390 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T01:54:46.390 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T01:54:46.393 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T01:54:46.402 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T01:54:46.405 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T01:54:46.405 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T01:54:46.407 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T01:54:46.408 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T01:54:46.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T01:54:46.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T01:54:46.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T01:54:46.411 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T01:54:46.411 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T01:54:46.411 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T01:54:46.412 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T01:54:46.412 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T01:54:46.413 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T01:54:46.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T01:54:46.433 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T01:54:46.436 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T01:54:46.438 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T01:54:46.440 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T01:54:46.441 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T01:54:46.445 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T01:54:46.448 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T01:54:46.464 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T01:54:46.467 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T01:54:46.517 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T01:54:46.518 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T01:54:46.518 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T01:54:46.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T01:54:46.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T01:54:46.520 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T01:54:46.526 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T01:54:46.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T01:54:46.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T01:54:46.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T01:54:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T01:54:46.537 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T01:54:46.558 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T01:54:46.559 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T01:54:46.560 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T01:54:46.560 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T01:54:46.562 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T01:54:46.563 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T01:54:46.578 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T01:54:46.591 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T01:54:46.594 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T01:54:46.597 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T01:54:46.613 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T01:54:46.718 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T01:54:46.723 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T01:54:46.753 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T01:54:46.894 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T01:54:46.902 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T01:54:46.926 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T01:54:47.097 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T01:54:47.346 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T01:54:47.540 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T01:54:47.559 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T01:54:47.573 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T01:54:47.616 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T01:54:47.626 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T01:54:47.633 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T01:54:47.637 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T01:54:47.641 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T01:54:47.643 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T01:54:47.657 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T01:54:47.686 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T01:54:47.821 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T01:54:47.868 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T01:54:47.876 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T01:54:47.881 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T01:54:47.904 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T01:54:47.944 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T01:54:48.089 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T01:54:48.123 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T01:54:48.234 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T01:54:48.344 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T01:54:48.345 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T01:54:48.348 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T01:54:48.353 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T01:54:48.374 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T01:54:48.435 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T01:54:48.440 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T01:54:48.466 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T01:54:48.474 INFO:teuthology.orchestra.run.smithi151.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi151.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi151.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T01:54:48.941 INFO:teuthology.orchestra.run.smithi151.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T01:54:48.973 INFO:teuthology.orchestra.run.smithi151.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T01:54:48.977 INFO:teuthology.orchestra.run.smithi151.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T01:54:49.026 INFO:teuthology.orchestra.run.smithi151.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T01:54:49.058 INFO:teuthology.orchestra.run.smithi151.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T01:54:49.085 INFO:teuthology.orchestra.run.smithi151.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T01:54:49.088 INFO:teuthology.orchestra.run.smithi151.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T01:54:49.096 INFO:teuthology.orchestra.run.smithi151.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T01:54:49.230 INFO:teuthology.orchestra.run.smithi151.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T01:54:49.242 INFO:teuthology.orchestra.run.smithi151.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T01:54:49.297 INFO:teuthology.orchestra.run.smithi151.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T01:54:49.533 INFO:teuthology.orchestra.run.smithi151.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T01:54:49.607 INFO:teuthology.orchestra.run.smithi151.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T01:54:49.728 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T01:54:49.728 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T01:54:49.742 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T01:54:49.758 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T01:54:49.759 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T01:54:49.760 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T01:54:49.774 INFO:teuthology.orchestra.run.smithi151.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T01:54:49.844 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T01:54:50.439 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T01:54:50.439 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T01:54:50.463 INFO:teuthology.orchestra.run.smithi151.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T01:54:50.464 INFO:teuthology.orchestra.run.smithi151.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T01:54:50.465 INFO:teuthology.orchestra.run.smithi151.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T01:54:50.466 INFO:teuthology.orchestra.run.smithi151.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T01:54:50.467 INFO:teuthology.orchestra.run.smithi151.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T01:54:50.468 INFO:teuthology.orchestra.run.smithi151.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T01:54:50.469 INFO:teuthology.orchestra.run.smithi151.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T01:54:50.471 INFO:teuthology.orchestra.run.smithi151.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T01:54:50.472 INFO:teuthology.orchestra.run.smithi151.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T01:54:50.544 INFO:teuthology.orchestra.run.smithi151.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T01:54:50.719 INFO:teuthology.orchestra.run.smithi151.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T01:54:51.099 INFO:teuthology.orchestra.run.smithi151.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T01:54:51.151 INFO:teuthology.orchestra.run.smithi151.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T01:54:51.186 INFO:teuthology.orchestra.run.smithi151.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T01:54:51.286 INFO:teuthology.orchestra.run.smithi151.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T01:54:51.334 INFO:teuthology.orchestra.run.smithi151.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T01:54:51.488 INFO:teuthology.orchestra.run.smithi151.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T01:54:51.498 INFO:teuthology.orchestra.run.smithi151.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T01:54:51.646 INFO:teuthology.orchestra.run.smithi151.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T01:54:51.648 INFO:teuthology.orchestra.run.smithi151.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T01:54:51.648 INFO:teuthology.orchestra.run.smithi151.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T01:54:51.812 INFO:teuthology.orchestra.run.smithi151.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T01:54:51.896 INFO:teuthology.orchestra.run.smithi151.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T01:54:52.077 INFO:teuthology.orchestra.run.smithi151.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T01:54:52.407 INFO:teuthology.orchestra.run.smithi151.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T01:54:52.409 INFO:teuthology.orchestra.run.smithi151.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T01:54:52.413 INFO:teuthology.orchestra.run.smithi151.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T01:54:52.414 INFO:teuthology.orchestra.run.smithi151.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T01:54:52.424 INFO:teuthology.orchestra.run.smithi151.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T01:54:52.429 INFO:teuthology.orchestra.run.smithi151.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T01:54:52.430 INFO:teuthology.orchestra.run.smithi151.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T01:54:52.431 INFO:teuthology.orchestra.run.smithi151.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T01:54:52.606 INFO:teuthology.orchestra.run.smithi151.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T01:54:52.768 INFO:teuthology.orchestra.run.smithi151.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T01:54:52.862 INFO:teuthology.orchestra.run.smithi151.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T01:54:53.544 INFO:teuthology.orchestra.run.smithi151.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T01:54:53.555 INFO:teuthology.orchestra.run.smithi151.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T01:54:53.620 INFO:teuthology.orchestra.run.smithi151.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T01:54:53.811 INFO:teuthology.orchestra.run.smithi151.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T01:54:54.908 INFO:teuthology.orchestra.run.smithi151.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T01:54:58.465 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T01:54:58.469 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T01:54:58.470 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T01:54:59.269 INFO:teuthology.orchestra.run.smithi151.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T01:54:59.270 INFO:teuthology.orchestra.run.smithi151.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T01:54:59.319 INFO:teuthology.orchestra.run.smithi151.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T01:54:59.666 INFO:teuthology.orchestra.run.smithi151.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T01:54:59.669 INFO:teuthology.orchestra.run.smithi151.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T01:54:59.707 INFO:teuthology.orchestra.run.smithi151.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T01:54:59.967 INFO:teuthology.orchestra.run.smithi151.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T01:55:00.971 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T01:55:01.072 INFO:teuthology.orchestra.run.smithi151.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T01:55:01.072 INFO:teuthology.orchestra.run.smithi151.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T01:55:01.592 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 209 MB in 15s (13.8 MB/s) 2023-09-30T01:55:01.735 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T01:55:01.794 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T01:55:01.798 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T01:55:01.892 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:55:02.238 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:02.364 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:55:02.986 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:03.111 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:55:03.824 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T01:55:03.840 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:03.882 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:04.368 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2023-09-30T01:55:04.385 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:04.418 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:04.804 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:04.862 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:05.115 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T01:55:05.132 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:05.181 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:05.501 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T01:55:05.519 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:05.584 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:05.845 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T01:55:05.861 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T01:55:05.895 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T01:55:06.123 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T01:55:06.140 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:06.181 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:06.534 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T01:55:06.550 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T01:55:06.609 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:55:06.953 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T01:55:06.971 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T01:55:07.012 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:55:07.281 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T01:55:07.297 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T01:55:07.331 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:55:07.686 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T01:55:07.705 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T01:55:07.736 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:55:08.047 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T01:55:08.064 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T01:55:08.105 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:55:08.108 INFO:teuthology.orchestra.run.smithi151.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T01:55:08.113 INFO:teuthology.orchestra.run.smithi151.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T01:55:08.114 INFO:teuthology.orchestra.run.smithi151.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T01:55:08.564 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T01:55:08.580 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:08.650 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:09.096 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T01:55:09.107 INFO:teuthology.orchestra.run.smithi151.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T01:55:09.112 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:09.146 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:09.488 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T01:55:09.505 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:09.538 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:09.744 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 209 MB in 23s (9,039 kB/s) 2023-09-30T01:55:09.914 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T01:55:09.924 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T01:55:09.940 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T01:55:09.970 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T01:55:09.974 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T01:55:09.982 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:55:10.064 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:55:10.328 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2023-09-30T01:55:10.344 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:10.371 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:10.386 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:10.496 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:55:11.105 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T01:55:11.121 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:11.155 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:11.192 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:11.327 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:55:11.482 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0. 2023-09-30T01:55:11.500 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T01:55:11.540 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T01:55:11.877 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T01:55:11.893 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:11.935 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:12.048 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T01:55:12.065 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:12.106 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:12.339 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2023-09-30T01:55:12.354 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:12.388 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:12.435 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rados. 2023-09-30T01:55:12.453 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:12.485 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:12.854 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T01:55:12.875 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:12.913 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:13.166 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T01:55:13.184 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:13.224 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:13.521 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T01:55:13.539 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:13.579 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:13.900 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T01:55:13.919 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T01:55:13.976 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T01:55:14.228 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T01:55:14.245 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:14.286 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:14.606 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T01:55:14.624 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T01:55:14.681 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:55:15.010 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T01:55:15.028 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2023-09-30T01:55:15.052 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:15.068 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:55:15.104 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:15.430 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T01:55:15.448 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T01:55:15.488 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:55:15.859 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T01:55:15.878 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T01:55:15.917 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:55:16.048 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T01:55:16.064 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T01:55:16.106 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:55:16.280 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T01:55:16.296 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T01:55:16.339 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:55:16.443 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T01:55:16.460 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T01:55:16.493 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T01:55:16.778 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T01:55:16.795 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:16.837 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:16.837 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T01:55:16.856 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:16.906 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:17.210 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T01:55:17.225 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T01:55:17.276 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:55:17.448 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T01:55:17.465 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:17.507 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:17.599 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T01:55:17.618 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T01:55:17.653 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:55:17.853 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T01:55:17.871 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:55:17.911 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:55:17.998 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T01:55:18.014 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T01:55:18.056 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T01:55:18.326 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T01:55:18.342 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T01:55:18.384 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:55:18.721 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T01:55:18.739 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T01:55:18.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T01:55:19.032 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T01:55:19.049 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T01:55:19.082 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T01:55:19.090 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:55:19.100 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T01:55:19.139 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:55:19.442 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2023-09-30T01:55:19.455 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librgw2. 2023-09-30T01:55:19.460 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T01:55:19.471 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:19.509 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:55:19.510 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T01:55:20.313 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:20.355 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:20.630 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T01:55:20.645 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T01:55:20.689 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:55:20.709 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liboath0. 2023-09-30T01:55:20.725 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T01:55:20.790 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T01:55:20.990 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T01:55:21.008 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T01:55:21.048 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:55:21.096 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T01:55:21.115 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:21.155 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:21.343 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2023-09-30T01:55:21.363 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T01:55:21.401 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T01:55:21.625 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-common. 2023-09-30T01:55:21.644 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:21.683 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:21.754 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T01:55:21.770 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T01:55:21.812 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:55:22.115 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T01:55:22.135 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T01:55:22.173 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T01:55:22.543 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T01:55:22.562 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T01:55:22.629 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:55:22.946 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T01:55:22.966 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T01:55:23.004 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T01:55:23.298 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2023-09-30T01:55:23.316 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T01:55:23.358 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:55:23.685 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T01:55:23.702 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T01:55:23.743 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:55:24.037 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T01:55:24.055 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T01:55:24.096 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:55:24.407 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T01:55:24.427 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T01:55:24.432 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-base. 2023-09-30T01:55:24.449 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:24.465 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T01:55:24.500 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:24.776 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T01:55:24.793 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T01:55:24.826 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:55:25.122 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T01:55:25.138 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T01:55:25.172 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:55:25.533 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T01:55:25.537 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T01:55:25.549 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:25.555 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T01:55:25.596 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:55:25.617 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:25.936 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T01:55:25.952 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:25.984 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T01:55:25.994 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:26.003 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T01:55:26.034 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T01:55:26.330 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T01:55:26.348 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:26.388 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:26.489 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T01:55:26.507 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:26.547 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:26.752 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T01:55:26.771 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T01:55:26.810 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:55:27.148 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T01:55:27.165 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T01:55:27.206 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:55:27.577 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T01:55:27.593 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T01:55:27.635 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T01:55:27.692 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T01:55:27.711 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:27.750 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:27.947 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T01:55:27.964 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T01:55:28.005 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:55:28.343 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T01:55:28.358 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T01:55:28.401 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T01:55:28.696 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T01:55:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T01:55:28.754 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:55:29.084 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-mako. 2023-09-30T01:55:29.101 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T01:55:29.142 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:55:29.462 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T01:55:29.480 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T01:55:29.546 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:55:29.849 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T01:55:29.865 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T01:55:29.907 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:55:30.236 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webob. 2023-09-30T01:55:30.252 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T01:55:30.294 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T01:55:30.512 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2023-09-30T01:55:30.528 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:30.593 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:30.615 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T01:55:30.632 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T01:55:30.673 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:55:30.897 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T01:55:30.913 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:30.955 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:31.001 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T01:55:31.017 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T01:55:31.060 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T01:55:31.384 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T01:55:31.403 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:31.422 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T01:55:31.440 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T01:55:31.441 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:31.483 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:55:31.826 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T01:55:31.842 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T01:55:31.884 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T01:55:31.953 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2023-09-30T01:55:31.970 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:32.011 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:32.216 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-paste. 2023-09-30T01:55:32.235 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T01:55:32.279 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:55:32.339 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T01:55:32.356 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T01:55:32.397 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T01:55:32.600 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T01:55:32.619 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T01:55:32.658 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:55:32.741 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T01:55:32.757 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:32.799 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:32.986 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T01:55:33.004 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T01:55:33.045 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:55:33.144 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T01:55:33.160 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T01:55:33.202 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T01:55:33.349 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T01:55:33.368 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T01:55:33.407 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T01:55:33.455 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T01:55:33.470 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T01:55:33.513 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:55:33.752 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T01:55:33.768 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T01:55:33.807 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T01:55:33.810 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:55:33.823 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T01:55:33.865 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:55:34.158 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T01:55:34.174 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T01:55:34.193 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2023-09-30T01:55:34.209 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T01:55:34.241 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:55:34.251 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T01:55:34.604 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T01:55:34.620 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:34.662 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:34.870 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T01:55:34.888 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:34.945 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:35.174 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T01:55:35.195 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T01:55:35.232 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:55:35.274 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T01:55:35.292 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:35.324 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:35.560 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T01:55:35.579 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T01:55:35.618 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:55:35.853 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T01:55:35.869 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:35.911 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:36.262 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T01:55:36.280 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T01:55:36.320 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T01:55:37.007 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T01:55:37.023 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:37.057 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:37.621 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T01:55:37.638 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T01:55:37.679 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:55:38.207 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T01:55:38.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T01:55:38.265 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T01:55:38.644 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T01:55:38.661 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T01:55:38.701 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:55:39.435 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T01:55:39.455 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:39.492 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:39.802 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph. 2023-09-30T01:55:39.821 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:39.861 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:40.189 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T01:55:40.205 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:40.247 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:40.601 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T01:55:40.620 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:40.629 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T01:55:40.645 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T01:55:40.659 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:40.687 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T01:55:40.973 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T01:55:40.989 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T01:55:41.031 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:55:41.188 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephadm. 2023-09-30T01:55:41.210 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:41.246 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:41.359 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T01:55:41.375 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T01:55:41.425 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:55:41.541 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T01:55:41.558 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T01:55:41.591 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T01:55:41.762 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T01:55:41.780 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T01:55:41.820 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T01:55:41.903 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T01:55:41.923 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:41.953 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:42.106 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T01:55:42.122 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T01:55:42.164 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T01:55:42.332 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T01:55:42.352 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T01:55:42.390 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T01:55:42.484 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T01:55:42.501 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T01:55:42.575 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:55:42.643 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T01:55:42.663 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T01:55:42.702 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:55:42.878 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T01:55:42.895 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T01:55:42.936 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:55:43.005 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T01:55:43.022 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T01:55:43.063 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:55:43.273 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T01:55:43.289 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T01:55:43.331 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:55:43.383 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-routes. 2023-09-30T01:55:43.403 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T01:55:43.441 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T01:55:43.650 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T01:55:43.669 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T01:55:43.708 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:55:43.820 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T01:55:43.837 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:43.878 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:44.028 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T01:55:44.045 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T01:55:44.086 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:55:44.416 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T01:55:44.433 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T01:55:44.474 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:55:44.624 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T01:55:44.643 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:44.682 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:44.777 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T01:55:44.793 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T01:55:44.876 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:55:45.046 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T01:55:45.062 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T01:55:45.104 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:55:45.239 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T01:55:45.259 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T01:55:45.297 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T01:55:45.440 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T01:55:45.459 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:55:45.523 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:55:45.801 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2023-09-30T01:55:45.817 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:55:45.867 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:55:46.120 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2023-09-30T01:55:46.139 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T01:55:46.178 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T01:55:46.532 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T01:55:46.542 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T01:55:46.548 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T01:55:46.559 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T01:55:46.590 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:55:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:55:46.936 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2023-09-30T01:55:46.953 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:46.994 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:47.179 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T01:55:47.195 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T01:55:47.237 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T01:55:47.575 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T01:55:47.594 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T01:55:47.633 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:55:48.427 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T01:55:48.445 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:48.485 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:49.639 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T01:55:49.657 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T01:55:49.730 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T01:55:50.033 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T01:55:50.052 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T01:55:50.092 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:55:50.446 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T01:55:50.462 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T01:55:50.504 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:55:50.816 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T01:55:50.833 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T01:55:50.874 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T01:55:51.219 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T01:55:51.236 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T01:55:51.277 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T01:55:51.600 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T01:55:51.618 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T01:55:51.658 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:55:51.986 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T01:55:52.003 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T01:55:52.044 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:55:52.056 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T01:55:52.075 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:52.114 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T01:55:52.416 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T01:55:52.454 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T01:55:52.455 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:55:52.472 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:55:52.504 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:55:52.766 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T01:55:52.782 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T01:55:52.824 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:55:53.024 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T01:55:53.043 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:53.082 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:53.169 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T01:55:53.186 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T01:55:53.227 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:55:53.469 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T01:55:53.485 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T01:55:53.544 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:55:53.881 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T01:55:53.886 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T01:55:53.899 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:55:53.904 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:55:53.939 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:53.953 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:55:54.353 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T01:55:54.370 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T01:55:54.436 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:55:54.789 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T01:55:54.806 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:55:54.847 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:55:55.150 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package jq. 2023-09-30T01:55:55.169 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:55:55.208 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:55:55.461 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package socat. 2023-09-30T01:55:55.480 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T01:55:55.519 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T01:55:55.581 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T01:55:55.599 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T01:55:55.664 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T01:55:55.881 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T01:55:55.897 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T01:55:55.939 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:55:56.001 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T01:55:56.017 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T01:55:56.059 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:55:56.285 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-test. 2023-09-30T01:55:56.302 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:55:56.343 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:55:56.362 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-attr. 2023-09-30T01:55:56.380 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T01:55:56.420 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T01:55:56.757 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T01:55:56.776 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T01:55:56.815 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:55:57.118 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T01:55:57.134 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T01:55:57.176 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T01:55:57.512 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T01:55:57.529 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T01:55:57.570 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:55:58.040 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T01:55:58.057 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T01:55:58.098 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T01:55:58.401 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T01:55:58.417 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T01:55:58.459 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T01:55:58.804 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T01:55:58.821 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T01:55:58.862 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T01:55:59.148 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T01:55:59.165 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:55:59.526 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2023-09-30T01:55:59.543 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T01:55:59.584 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:55:59.929 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T01:55:59.945 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T01:55:59.978 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:56:00.290 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T01:56:00.308 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T01:56:00.347 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:56:00.671 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T01:56:00.691 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T01:56:00.733 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T01:56:01.070 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T01:56:01.087 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T01:56:01.129 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:56:01.323 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T01:56:01.338 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:56:01.380 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:01.406 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T01:56:01.424 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T01:56:01.472 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:56:01.693 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T01:56:01.713 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:56:01.751 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:02.210 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2023-09-30T01:56:02.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:56:02.268 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:02.288 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T01:56:02.306 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:56:02.346 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:02.744 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T01:56:02.762 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T01:56:02.820 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:56:03.149 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T01:56:03.169 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:56:03.208 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:03.996 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T01:56:04.016 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:56:04.054 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:04.360 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T01:56:04.676 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:56:04.802 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:56:04.910 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T01:56:04.928 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T01:56:04.928 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:56:04.967 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T01:56:05.122 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:05.305 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T01:56:05.325 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T01:56:05.732 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:56:05.791 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2023-09-30T01:56:05.954 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:06.065 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:56:06.149 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-attr. 2023-09-30T01:56:06.169 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T01:56:06.207 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T01:56:06.409 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T01:56:06.535 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T01:56:06.551 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T01:56:06.630 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:56:06.793 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:06.937 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T01:56:06.954 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T01:56:06.995 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T01:56:07.344 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T01:56:07.357 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T01:56:07.374 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T01:56:07.415 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:56:07.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:56:07.796 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T01:56:07.851 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T01:56:07.870 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T01:56:07.909 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T01:56:08.207 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:56:08.237 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T01:56:08.254 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T01:56:08.295 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T01:56:08.500 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T01:56:08.648 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T01:56:08.669 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T01:56:08.706 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T01:56:08.784 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:56:09.026 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T01:56:09.043 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T01:56:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:56:09.084 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:56:09.362 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:09.445 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-py. 2023-09-30T01:56:09.462 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T01:56:09.503 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:56:09.671 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:56:09.805 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:56:09.831 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T01:56:09.852 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T01:56:09.889 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:56:10.090 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:56:10.225 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T01:56:10.242 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T01:56:10.283 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:56:10.349 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:56:10.475 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:56:10.586 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T01:56:10.603 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T01:56:10.677 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T01:56:10.838 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T01:56:11.031 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T01:56:11.048 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T01:56:11.089 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:56:11.139 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:56:11.383 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T01:56:11.404 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T01:56:11.415 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:56:11.441 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:56:11.750 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:56:12.034 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T01:56:12.205 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package radosgw. 2023-09-30T01:56:12.222 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:56:12.262 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:12.368 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T01:56:12.677 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T01:56:12.973 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:56:13.090 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:56:13.216 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:56:13.500 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:56:13.902 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:56:13.966 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T01:56:13.985 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:56:14.048 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:14.303 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:56:14.347 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T01:56:14.637 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:56:14.688 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T01:56:14.788 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:56:14.814 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:14.914 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:56:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:15.140 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:15.275 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T01:56:15.644 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:56:15.730 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user cephadm....done 2023-09-30T01:56:15.777 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:56:15.891 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:15.903 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T01:56:16.051 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:56:16.195 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:56:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T01:56:16.505 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:16.666 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:56:16.737 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:56:16.792 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:56:16.863 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T01:56:16.989 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:56:17.127 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:56:17.307 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T01:56:17.428 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T01:56:17.721 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:56:17.742 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:56:17.830 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:56:18.035 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T01:56:18.124 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T01:56:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:56:18.424 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:56:18.621 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:56:18.742 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:56:18.914 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:19.168 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T01:56:19.198 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:56:19.316 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:56:19.414 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:56:19.540 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:56:19.641 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:56:19.875 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:56:19.932 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:56:20.049 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:56:20.226 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:56:20.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:56:20.442 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T01:56:20.478 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T01:56:20.727 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:56:20.804 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:56:21.036 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:56:21.092 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:21.218 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T01:56:21.372 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:56:21.689 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T01:56:22.032 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T01:56:22.351 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T01:56:22.644 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:56:22.682 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:22.795 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:56:22.808 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:56:22.930 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:56:22.936 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T01:56:23.231 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:56:23.313 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T01:56:23.614 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:56:23.615 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:56:23.915 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:23.984 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:56:24.040 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T01:56:24.352 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T01:56:24.375 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:24.478 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:24.501 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:56:24.829 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:24.845 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:24.964 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T01:56:24.970 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:56:25.305 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:56:25.323 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:56:25.474 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:56:25.600 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T01:56:25.885 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:56:26.203 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:26.328 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:56:26.335 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:26.461 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:56:26.463 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:56:26.612 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T01:56:26.713 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:56:26.773 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:56:26.840 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T01:56:27.091 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T01:56:27.140 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:56:27.275 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T01:56:27.392 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:56:27.551 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:56:27.593 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:27.718 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:27.844 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T01:56:27.939 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:56:27.940 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:56:28.151 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:56:28.420 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:28.465 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:56:28.657 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:56:28.868 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T01:56:28.949 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T01:56:29.118 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:56:29.244 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:56:29.267 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:56:29.562 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:56:29.620 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:29.746 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:29.872 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:29.914 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:56:29.998 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:30.064 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:56:30.124 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T01:56:30.227 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T01:56:30.425 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T01:56:30.553 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:56:30.743 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:30.861 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:30.871 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:30.987 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:56:30.997 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T01:56:31.890 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:32.434 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2023-09-30T01:56:32.438 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:32.556 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:56:32.681 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T01:56:32.819 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2023-09-30T01:56:32.995 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2023-09-30T01:56:33.019 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T01:56:33.058 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T01:56:33.168 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T01:56:33.359 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:56:33.485 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T01:56:33.652 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:56:33.778 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T01:56:33.942 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:34.067 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:34.104 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:34.263 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:56:34.334 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:56:34.334 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:56:34.599 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:34.716 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:56:34.952 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:35.042 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:56:35.303 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T01:56:35.771 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:36.012 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:56:36.012 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:56:36.074 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:36.229 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:56:36.350 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T01:56:36.472 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:56:36.722 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:56:36.722 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:56:36.912 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T01:56:37.174 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:37.261 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T01:56:37.686 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:56:37.687 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:56:37.688 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:37.814 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:38.034 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:56:38.035 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:56:38.284 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:38.402 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:38.490 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:38.561 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:38.702 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:56:38.781 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:56:38.781 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:56:39.011 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T01:56:39.212 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.330 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:56:39.355 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.481 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.657 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.699 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.785 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:56:39.825 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:39.938 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T01:56:39.951 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:40.077 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:40.211 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T01:56:40.240 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:56:40.538 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T01:56:40.831 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:40.957 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:41.108 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:56:42.036 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:42.597 INFO:teuthology.orchestra.run.smithi151.stdout:Adding group ceph....done 2023-09-30T01:56:43.014 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user ceph....done 2023-09-30T01:56:43.042 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T01:56:43.192 INFO:teuthology.orchestra.run.smithi151.stdout:Setting system user ceph properties....done 2023-09-30T01:56:43.213 INFO:teuthology.orchestra.run.smithi151.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T01:56:43.360 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T01:56:43.672 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T01:56:43.924 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:56:43.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install sqlite3 2023-09-30T01:56:43.997 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2023-09-30T01:56:44.130 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2023-09-30T01:56:44.256 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:44.386 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:56:44.387 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:56:44.387 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:56:44.388 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2023-09-30T01:56:44.389 INFO:teuthology.orchestra.run.smithi172.stdout: libsqlite3-0 2023-09-30T01:56:44.390 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2023-09-30T01:56:44.390 INFO:teuthology.orchestra.run.smithi172.stdout: sqlite3-doc 2023-09-30T01:56:44.425 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2023-09-30T01:56:44.425 INFO:teuthology.orchestra.run.smithi172.stdout: sqlite3 2023-09-30T01:56:44.426 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi172.stdout: libsqlite3-0 2023-09-30T01:56:44.557 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:56:44.557 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:56:44.629 INFO:teuthology.orchestra.run.smithi172.stdout:1 upgraded, 1 newly installed, 0 to remove and 244 not upgraded. 2023-09-30T01:56:44.629 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 1,409 kB of archives. 2023-09-30T01:56:44.629 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 2,803 kB of additional disk space will be used. 2023-09-30T01:56:44.629 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.5 [549 kB] 2023-09-30T01:56:45.177 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 sqlite3 amd64 3.31.1-4ubuntu0.5 [860 kB] 2023-09-30T01:56:45.199 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:45.553 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T01:56:45.650 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 1,409 kB in 1s (1,645 kB/s) 2023-09-30T01:56:45.885 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152614 files and directories currently installed.) 2023-09-30T01:56:45.889 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../libsqlite3-0_3.31.1-4ubuntu0.5_amd64.deb ... 2023-09-30T01:56:46.017 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:46.064 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) over (3.31.1-4ubuntu0.3) ... 2023-09-30T01:56:46.256 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:56:46.257 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:56:46.427 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package sqlite3. 2023-09-30T01:56:46.444 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../sqlite3_3.31.1-4ubuntu0.5_amd64.deb ... 2023-09-30T01:56:46.478 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking sqlite3 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:46.897 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:46.981 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:56:46.981 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:56:47.025 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up sqlite3 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:47.151 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:56:47.637 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:56:47.655 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:48.079 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:56:48.081 DEBUG:teuthology.parallel:result is None 2023-09-30T01:56:48.180 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:56:48.181 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:56:48.623 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:48.749 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:48.892 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:49.119 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:56:49.119 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:56:49.544 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:49.673 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:49.832 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:49.975 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:56:50.102 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:56:50.253 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T01:56:50.538 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:56:53.500 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T01:56:54.109 INFO:teuthology.orchestra.run.smithi151.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:56:54.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install sqlite3 2023-09-30T01:56:54.179 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2023-09-30T01:56:54.562 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:56:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:56:54.563 INFO:teuthology.orchestra.run.smithi151.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:56:54.563 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2023-09-30T01:56:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: libsqlite3-0 2023-09-30T01:56:54.565 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2023-09-30T01:56:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: sqlite3-doc 2023-09-30T01:56:54.599 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2023-09-30T01:56:54.599 INFO:teuthology.orchestra.run.smithi151.stdout: sqlite3 2023-09-30T01:56:54.600 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2023-09-30T01:56:54.601 INFO:teuthology.orchestra.run.smithi151.stdout: libsqlite3-0 2023-09-30T01:56:54.653 INFO:teuthology.orchestra.run.smithi151.stdout:1 upgraded, 1 newly installed, 0 to remove and 244 not upgraded. 2023-09-30T01:56:54.653 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 1,409 kB of archives. 2023-09-30T01:56:54.653 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 2,803 kB of additional disk space will be used. 2023-09-30T01:56:54.653 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.5 [549 kB] 2023-09-30T01:56:54.766 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 sqlite3 amd64 3.31.1-4ubuntu0.5 [860 kB] 2023-09-30T01:56:55.156 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 1,409 kB in 0s (7,809 kB/s) 2023-09-30T01:56:55.382 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152614 files and directories currently installed.) 2023-09-30T01:56:55.387 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../libsqlite3-0_3.31.1-4ubuntu0.5_amd64.deb ... 2023-09-30T01:56:55.556 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) over (3.31.1-4ubuntu0.3) ... 2023-09-30T01:56:55.927 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package sqlite3. 2023-09-30T01:56:55.945 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../sqlite3_3.31.1-4ubuntu0.5_amd64.deb ... 2023-09-30T01:56:55.984 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking sqlite3 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:56.394 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:56.548 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up sqlite3 (3.31.1-4ubuntu0.5) ... 2023-09-30T01:56:56.674 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:56:57.195 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:56:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:56:57.908 DEBUG:teuthology.parallel:result is None 2023-09-30T01:56:57.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:56:58.060 DEBUG:teuthology.orchestra.run.smithi151:> dpkg-query -W -f '${Version}' ceph 2023-09-30T01:56:58.084 INFO:teuthology.orchestra.run.smithi151.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:56:58.084 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:56:58.085 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T01:56:58.085 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:56:58.231 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2023-09-30T01:56:58.255 INFO:teuthology.orchestra.run.smithi172.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:56:58.255 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:56:58.255 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T01:56:58.255 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T01:56:58.256 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:58.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:56:58.273 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:58.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:56:58.310 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T01:56:58.310 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:58.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:56:58.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:56:58.344 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:58.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:56:58.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:56:58.374 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T01:56:58.374 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:58.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:56:58.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:56:58.400 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:58.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:56:58.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:56:58.471 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T01:56:58.471 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:58.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:56:58.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:56:58.503 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:58.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:56:58.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:56:58.578 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T01:56:58.645 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T01:56:58.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2023-09-30T01:56:58.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-09-30T01:56:58.665 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T01:56:58.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:56:58.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:56:58.728 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T01:56:58.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:56:58.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:56:58.758 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T01:56:58.758 INFO:tasks.ceph:config {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T01:56:58.758 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7407135', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, {'ceph': None}, {'exec': {'client.0': ['ceph osd pool create cephsqlite', "ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring"]}}, {'exec': {'client.0': ['ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr']}}, {'workunit': {'clients': {'client.0': ['rados/test_libcephsqlite.sh cephsqlite']}, 'env': {'CEPH_ARGS': '--id libcephsqlite --no-log-to-stderr'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T01:56:58.759 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:56:58.789 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:56:58.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:56:58.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:56:58.851 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:58.851 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:56:58.861 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:56:58.862 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-09-30 01:53:08.480311909 +0000 2023-09-30T01:56:58.868 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-09-30 01:53:08.476312002 +0000 2023-09-30T01:56:58.869 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-09-30 01:53:08.476312002 +0000 2023-09-30T01:56:58.869 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-09-30T01:56:58.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:56:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-09-30T01:56:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-09-30T01:56:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000399361 s, 1.3 MB/s 2023-09-30T01:56:58.883 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:56:58.894 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-09-30 01:53:08.976300482 +0000 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-09-30 01:53:08.972300576 +0000 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-09-30 01:53:08.972300576 +0000 2023-09-30T01:56:58.946 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-09-30T01:56:58.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:56:58.958 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-09-30T01:56:58.958 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-09-30T01:56:58.958 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000415301 s, 1.2 MB/s 2023-09-30T01:56:58.959 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:56:58.968 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2023-09-30T01:56:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-09-30 01:53:09.528287766 +0000 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-09-30 01:53:09.524287857 +0000 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-09-30 01:53:09.524287857 +0000 2023-09-30T01:56:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-09-30T01:56:58.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:56:58.991 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-09-30T01:56:58.991 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-09-30T01:56:58.991 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000387452 s, 1.3 MB/s 2023-09-30T01:56:58.991 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:56:59.043 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-09-30 01:53:10.304269887 +0000 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-09-30 01:53:10.028276246 +0000 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-09-30 01:53:10.028276246 +0000 2023-09-30T01:56:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2023-09-30T01:56:59.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:56:59.062 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-09-30T01:56:59.063 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-09-30T01:56:59.063 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000411577 s, 1.2 MB/s 2023-09-30T01:56:59.065 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:56:59.115 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:56:59.115 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:59.115 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:56:59.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:56:59.122 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-09-30T01:56:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:56:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:59.172 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-09-30T01:56:59.172 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:59.172 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-09-30 01:52:58.723585780 +0000 2023-09-30T01:56:59.172 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-09-30 01:52:58.443592223 +0000 2023-09-30T01:56:59.172 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-09-30 01:52:58.443592223 +0000 2023-09-30T01:56:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-09-30T01:56:59.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:56:59.184 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-09-30T01:56:59.184 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-09-30T01:56:59.184 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000397402 s, 1.3 MB/s 2023-09-30T01:56:59.185 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:56:59.194 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-09-30T01:56:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:56:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-09-30 01:52:59.231574089 +0000 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-09-30 01:52:58.959580349 +0000 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-09-30 01:52:58.959580349 +0000 2023-09-30T01:56:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-09-30T01:56:59.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:56:59.257 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-09-30T01:56:59.258 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-09-30T01:56:59.258 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000398339 s, 1.3 MB/s 2023-09-30T01:56:59.259 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:56:59.266 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-09-30T01:56:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:56:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:59.272 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 373 Links: 1 2023-09-30T01:56:59.273 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:59.273 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-09-30 01:52:59.763561847 +0000 2023-09-30T01:56:59.273 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-09-30 01:52:59.471568567 +0000 2023-09-30T01:56:59.273 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-09-30 01:52:59.471568567 +0000 2023-09-30T01:56:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-09-30T01:56:59.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:56:59.329 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-09-30T01:56:59.329 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-09-30T01:56:59.329 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000379546 s, 1.3 MB/s 2023-09-30T01:56:59.330 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:56:59.339 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-09-30T01:56:59.344 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:56:59.344 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:56:59.344 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 376 Links: 1 2023-09-30T01:56:59.345 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:56:59.345 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-09-30 01:52:59.991556600 +0000 2023-09-30T01:56:59.345 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-09-30 01:52:59.987556693 +0000 2023-09-30T01:56:59.345 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-09-30 01:52:59.987556693 +0000 2023-09-30T01:56:59.345 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-09-30T01:56:59.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:56:59.357 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-09-30T01:56:59.358 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-09-30T01:56:59.358 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000413934 s, 1.2 MB/s 2023-09-30T01:56:59.359 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:56:59.409 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:56:59.409 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T01:56:59.409 INFO:tasks.ceph:Generating config... 2023-09-30T01:56:59.410 INFO:tasks.ceph:[client] debug cephsqlite = 20 2023-09-30T01:56:59.411 INFO:tasks.ceph:[client] debug client = 20 2023-09-30T01:56:59.411 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T01:56:59.411 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T01:56:59.411 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T01:56:59.411 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T01:56:59.411 INFO:tasks.ceph:[global] ms type = async 2023-09-30T01:56:59.411 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T01:56:59.411 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T01:56:59.411 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T01:56:59.411 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T01:56:59.411 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T01:56:59.412 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T01:56:59.412 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-09-30T01:56:59.412 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-09-30T01:56:59.412 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T01:56:59.412 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T01:56:59.412 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T01:56:59.413 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T01:56:59.414 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T01:56:59.414 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T01:56:59.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T01:56:59.448 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T01:56:59.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T01:56:59.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:56:59.538 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.151'), ('mon.c', '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]'), ('mon.b', '172.21.15.172')] 2023-09-30T01:56:59.538 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.172', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T01:56:59.539 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:59.539 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:56:59.544 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.151 --addv c '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]' --add b 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:56:59.583 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:56:59.583 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: generated fsid 70a77e99-16f3-4896-b121-bcea9198ad08 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = octopus 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 0 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 70a77e99-16f3-4896-b121-bcea9198ad08 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:last_changed 2023-09-30T01:56:59.583257+0000 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:created 2023-09-30T01:56:59.583257+0000 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:min_mon_release 15 (octopus) 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:election_strategy: 1 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-09-30T01:56:59.584 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T01:56:59.586 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:56:59.633 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 70a77e99-16f3-4896-b121-bcea9198ad08... 2023-09-30T01:56:59.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:56:59.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:56:59.671 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2023-09-30T01:56:59.671 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2023-09-30T01:56:59.671 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.675 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2023-09-30T01:56:59.676 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.677 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2023-09-30T01:56:59.678 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.678 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2023-09-30T01:56:59.678 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.678 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2023-09-30T01:56:59.679 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2023-09-30T01:56:59.679 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2023-09-30T01:56:59.679 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.172" 2023-09-30T01:56:59.679 INFO:teuthology.orchestra.run.smithi151.stdout: mon client directed command retry = 5 2023-09-30T01:56:59.679 INFO:teuthology.orchestra.run.smithi151.stdout: mon election default strategy = 1 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: ms inject socket failures = 5000 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: ms type = async 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 70a77e99-16f3-4896-b121-bcea9198ad08 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2023-09-30T01:56:59.680 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2023-09-30T01:56:59.681 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2023-09-30T01:56:59.682 INFO:teuthology.orchestra.run.smithi151.stdout: bluefs allocator = hybrid 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore allocator = hybrid 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore zero block detection = True 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 20 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 20 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2023-09-30T01:56:59.683 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify cached snaps = True 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify missing on start = True 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock override recovery settings = True 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue = debug_random 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue cut off = debug_random 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: osd shutdown pgref assert = True 2023-09-30T01:56:59.684 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.685 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2023-09-30T01:56:59.685 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-09-30T01:56:59.685 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2023-09-30T01:56:59.685 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-09-30T01:56:59.685 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2023-09-30T01:56:59.686 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2023-09-30T01:56:59.687 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:56:59.688 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.689 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2023-09-30T01:56:59.690 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:56:59.690 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.690 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:56:59.690 INFO:teuthology.orchestra.run.smithi151.stdout: mon min osdmap epochs = 50 2023-09-30T01:56:59.690 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune interval = 2 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune min = 15 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout: mon scrub interval = 300 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout: paxos service trim min = 10 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:56:59.691 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: debug cephsqlite = 20 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: debug client = 20 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2023-09-30T01:56:59.692 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2023-09-30T01:56:59.693 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2023-09-30T01:56:59.694 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-09-30T01:56:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-09-30T01:56:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:56:59.696 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:56:59.697 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:56:59.698 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-09-30T01:56:59.698 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:56:59.698 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-09-30T01:56:59.698 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:56:59.698 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-09-30T01:56:59.699 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.700 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-09-30T01:56:59.700 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.700 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-09-30T01:56:59.700 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-09-30T01:56:59.700 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-09-30T01:56:59.701 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.172" 2023-09-30T01:56:59.701 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2023-09-30T01:56:59.701 INFO:teuthology.orchestra.run.smithi172.stdout: mon election default strategy = 1 2023-09-30T01:56:59.701 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout: ms type = async 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 70a77e99-16f3-4896-b121-bcea9198ad08 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-09-30T01:56:59.702 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-09-30T01:56:59.703 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.704 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-09-30T01:56:59.704 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-09-30T01:56:59.704 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-09-30T01:56:59.704 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.704 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-09-30T01:56:59.705 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.706 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-09-30T01:56:59.706 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.706 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:56:59.706 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-09-30T01:56:59.706 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.707 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-09-30T01:56:59.707 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-09-30T01:56:59.707 INFO:teuthology.orchestra.run.smithi172.stdout: bluefs allocator = hybrid 2023-09-30T01:56:59.707 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = hybrid 2023-09-30T01:56:59.707 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore zero block detection = True 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-09-30T01:56:59.708 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2023-09-30T01:56:59.709 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:56:59.709 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2023-09-30T01:56:59.709 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:56:59.709 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify cached snaps = True 2023-09-30T01:56:59.709 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify missing on start = True 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock override recovery settings = True 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue = debug_random 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue cut off = debug_random 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-09-30T01:56:59.710 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-09-30T01:56:59.711 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-09-30T01:56:59.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-09-30T01:56:59.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: mon scrub interval = 300 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-09-30T01:56:59.714 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug cephsqlite = 20 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 20 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-09-30T01:56:59.715 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-09-30T01:56:59.716 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2023-09-30T01:56:59.716 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2023-09-30T01:56:59.716 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T01:56:59.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T01:56:59.782 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T01:56:59.782 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:59.782 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T01:56:59.787 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:59.787 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T01:56:59.793 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:56:59.793 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:59.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:56:59.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:56:59.817 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:56:59.817 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:56:59.865 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:56:59.865 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:59.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:56:59.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:56:59.893 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:56:59.893 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:56:59.943 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T01:56:59.943 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T01:56:59.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T01:56:59.991 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T01:56:59.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:57:00.036 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:57:00.039 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T01:57:00.039 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T01:57:00.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T01:57:00.076 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T01:57:00.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T01:57:00.126 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T01:57:00.140 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T01:57:00.140 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T01:57:00.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T01:57:00.156 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:00.156 INFO:tasks.ceph:role: osd.0 2023-09-30T01:57:00.156 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:57:00.156 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:57:00.684 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:00.685 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:00.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-09-30T01:57:00.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T01:57:00.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T01:57:00.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:00.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T01:57:00.815 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:00.815 INFO:tasks.ceph:role: osd.1 2023-09-30T01:57:00.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:57:00.815 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:57:01.316 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:01.317 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:01.319 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-09-30T01:57:01.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T01:57:01.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T01:57:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:01.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:01.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T01:57:01.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:01.402 INFO:tasks.ceph:role: osd.2 2023-09-30T01:57:01.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:57:01.402 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:01.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-09-30T01:57:01.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T01:57:01.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T01:57:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:02.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T01:57:02.063 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:02.063 INFO:tasks.ceph:role: osd.3 2023-09-30T01:57:02.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T01:57:02.063 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T01:57:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:02.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-09-30T01:57:02.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T01:57:02.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T01:57:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:02.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:02.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:02.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.978+0000 7fc354e533c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T01:57:02.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.978+0000 7fc354e533c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T01:57:02.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.978+0000 7fc354e533c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:57:02.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.978+0000 7fc354e533c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:57:02.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.978+0000 7fc354e533c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:57:02.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:02.986+0000 7fc354e533c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T01:57:03.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:03.794+0000 7fc354e533c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:04.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:04.574+0000 7fc354e533c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:05.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:05.357+0000 7fc354e533c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:06.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T01:57:06.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.273+0000 7f1ac9c6c3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T01:57:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.273+0000 7f1ac9c6c3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T01:57:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.273+0000 7f1ac9c6c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:57:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.273+0000 7f1ac9c6c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:57:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.273+0000 7f1ac9c6c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:57:06.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.277+0000 7f1ac9c6c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T01:57:06.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:06.841+0000 7f1ac9c6c3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:07.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:07.625+0000 7f1ac9c6c3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:08.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:08.413+0000 7f1ac9c6c3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:09.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T01:57:09.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:09.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T01:57:09.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T01:57:09.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:57:09.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:57:09.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:57:09.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.377+0000 7fe03fb943c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T01:57:09.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:09.929+0000 7fe03fb943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:10.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:10.717+0000 7fe03fb943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:11.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:11.505+0000 7fe03fb943c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:12.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T01:57:12.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:12.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.413+0000 7f96b58293c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T01:57:12.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:12.981+0000 7f96b58293c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:13.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:13.773+0000 7f96b58293c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:14.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:14.561+0000 7f96b58293c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:15.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T01:57:15.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T01:57:15.484 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:15.484 INFO:tasks.ceph:role: osd.4 2023-09-30T01:57:15.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:57:15.484 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:57:15.989 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:15.989 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:15.989 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:15.990 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:15.991 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:15.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-09-30T01:57:15.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T01:57:16.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T01:57:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:16.099 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:16.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T01:57:16.157 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:16.157 INFO:tasks.ceph:role: osd.5 2023-09-30T01:57:16.157 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:57:16.157 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:16.684 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:16.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-09-30T01:57:16.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T01:57:16.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T01:57:16.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:16.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T01:57:16.792 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:16.792 INFO:tasks.ceph:role: osd.6 2023-09-30T01:57:16.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:57:16.792 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:57:17.267 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:17.267 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:17.267 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:17.267 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:17.268 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:17.269 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-09-30T01:57:17.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T01:57:17.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T01:57:17.346 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:17.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T01:57:17.357 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:57:17.357 INFO:tasks.ceph:role: osd.7 2023-09-30T01:57:17.357 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T01:57:17.357 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T01:57:17.856 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:57:17.857 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:57:17.859 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-09-30T01:57:17.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T01:57:17.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T01:57:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:57:17.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:57:17.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:18.015 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.013+0000 7fba7ade93c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T01:57:18.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.017+0000 7fba7ade93c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T01:57:18.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.017+0000 7fba7ade93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:57:18.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.017+0000 7fba7ade93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:57:18.019 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.017+0000 7fba7ade93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:57:18.020 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.017+0000 7fba7ade93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T01:57:18.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:18.825+0000 7fba7ade93c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:19.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:19.609+0000 7fba7ade93c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:20.395 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:20.393+0000 7fba7ade93c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:21.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T01:57:21.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:21.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.289+0000 7f838c65f3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T01:57:21.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.289+0000 7f838c65f3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T01:57:21.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.289+0000 7f838c65f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:57:21.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.289+0000 7f838c65f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:57:21.292 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.289+0000 7f838c65f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:57:21.293 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.293+0000 7f838c65f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T01:57:21.849 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:21.845+0000 7f838c65f3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:22.630 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:22.629+0000 7f838c65f3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:23.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:23.421+0000 7f838c65f3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:24.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T01:57:24.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:24.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T01:57:24.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T01:57:24.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:57:24.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:57:24.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:57:24.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.381+0000 7f974f1063c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T01:57:24.942 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:24.941+0000 7f974f1063c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:25.737 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:25.737+0000 7f974f1063c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:26.527 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:26.525+0000 7f974f1063c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:27.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T01:57:27.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.484+0000 7f9b524673c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.484+0000 7f9b524673c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.484+0000 7f9b524673c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.484+0000 7f9b524673c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.484+0000 7f9b524673c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:57:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:27.488+0000 7f9b524673c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T01:57:28.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:28.048+0000 7f9b524673c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:28.828+0000 7f9b524673c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:29.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:29.608+0000 7f9b524673c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:57:30.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T01:57:30.476 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T01:57:30.476 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T01:57:30.492 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T01:57:30.551 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T01:57:30.560 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T01:57:30.574 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T01:57:30.629 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T01:57:30.645 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T01:57:30.703 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T01:57:30.714 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T01:57:30.727 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T01:57:30.737 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:30.737 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T01:57:30.744 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:30.744 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T01:57:30.749 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T01:57:30.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:57:30.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:57:30.795 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2023-09-30T01:57:30.795 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBrgBdlhCcVOxAAxU3vPFsWUzEhh5E0jE6kUQ== 2023-09-30T01:57:30.795 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBugBdl5U1eOhAAmy8BI9YH5gzQuncSi/uejA== 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBygBdl9PqYEBAAlK8hA/g0BWF5Y7me7WCdIA== 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB1gBdlFlCaFhAAzGMMDa79ZovvozUo82CjIA== 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB4gBdlayDhGBAA26S7xnf6ThelRFgRwmiacA== 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBsgBdlj3o+AhAA/ynLIQGEcXc3JKh+sk/C3A== 2023-09-30T01:57:30.796 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB+gBdlb9QBARAANTaNsnhWgf0kG5cMmR20sg== 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCBgBdl+Il7ERAA+Kk6REBXVrNeS+WO1Dv9mg== 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCEgBdlbiXPFhAAaYyQyZB0N6xoKct3M5RJ1A== 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCHgBdle9coHRAAdlCcbOyJqGONxeJiFG2dKg== 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBsgBdlz5uDBBAAGs76faLqSmwfqDTsIDIeLQ== 2023-09-30T01:57:30.797 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2023-09-30T01:57:30.798 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBsgBdlrNSgBxAA4LopmiNJOTNfO3PadKfrYQ== 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBrgBdlhCcVOxAAxU3vPFsWUzEhh5E0jE6kUQ== 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBugBdl5U1eOhAAmy8BI9YH5gzQuncSi/uejA== 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBygBdl9PqYEBAAlK8hA/g0BWF5Y7me7WCdIA== 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQB1gBdlFlCaFhAAzGMMDa79ZovvozUo82CjIA== 2023-09-30T01:57:30.804 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQB4gBdlayDhGBAA26S7xnf6ThelRFgRwmiacA== 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBsgBdlj3o+AhAA/ynLIQGEcXc3JKh+sk/C3A== 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQB+gBdlb9QBARAANTaNsnhWgf0kG5cMmR20sg== 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCBgBdl+Il7ERAA+Kk6REBXVrNeS+WO1Dv9mg== 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCEgBdlbiXPFhAAaYyQyZB0N6xoKct3M5RJ1A== 2023-09-30T01:57:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2023-09-30T01:57:30.806 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCHgBdle9coHRAAdlCcbOyJqGONxeJiFG2dKg== 2023-09-30T01:57:30.806 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-09-30T01:57:30.806 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBsgBdlz5uDBBAAGs76faLqSmwfqDTsIDIeLQ== 2023-09-30T01:57:30.806 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2023-09-30T01:57:30.806 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBsgBdlrNSgBxAA4LopmiNJOTNfO3PadKfrYQ== 2023-09-30T01:57:30.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:57:30.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:57:30.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:30.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:30.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:30.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:57:31.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:57:31.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:57:31.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:57:31.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:57:31.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:57:31.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:57:31.565 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T01:57:31.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T01:57:31.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:57:31.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T01:57:31.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T01:57:31.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:57:31.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T01:57:31.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T01:57:31.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:57:31.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T01:57:31.912 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:31.915 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:57:31.920 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T01:57:31.920 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T01:57:31.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T01:57:31.922 INFO:tasks.ceph.mon.a:Started 2023-09-30T01:57:31.923 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T01:57:31.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T01:57:31.925 INFO:tasks.ceph.mon.c:Started 2023-09-30T01:57:31.925 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T01:57:31.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T01:57:31.962 INFO:tasks.ceph.mon.b:Started 2023-09-30T01:57:31.962 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T01:57:31.963 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T01:57:31.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T01:57:31.965 INFO:tasks.ceph.mgr.y:Started 2023-09-30T01:57:31.965 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T01:57:31.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T01:57:31.967 INFO:tasks.ceph.mgr.x:Started 2023-09-30T01:57:31.967 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T01:57:31.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T01:57:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 Processor -- start 2023-09-30T01:57:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 -- start start 2023-09-30T01:57:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac06e230 con 0x7f0bac0ab5c0 2023-09-30T01:57:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac06df00 con 0x7f0bac06b480 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb29ed700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac06d800 con 0x7f0bac06b850 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bab7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb0f8a700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0bac0ab5c0 0x7f0bac144420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45204/0 (socket says 172.21.15.151:45204) 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bb0f8a700 1 -- 172.21.15.151:0/4199108043 learned_addr learned my addr 172.21.15.151:0/4199108043 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.057+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:57:32.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.261+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:32.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.261+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:32.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:32.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:57:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.665+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.665+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:32.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:57:33.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:33.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:33.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:33.469+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:33.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:33.469+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:33.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:33.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:33.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:33.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 msgr2=0x7f0bac06e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac06b850 0x7f0bac06e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 >> v1:172.21.15.172:6789/0 conn(0x7f0bac06b480 legacy=0x7f0bac139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 >> v1:172.21.15.151:6789/0 conn(0x7f0bac0ab5c0 legacy=0x7f0bac144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac06d800 con 0x7f0b980062a0 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b98008dd0 con 0x7f0b98003760 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.057+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:57:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:57:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:57:35.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:35.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.261+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:57:35.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:35.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:35.665+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:57:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:57:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:57:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:57:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0babfff700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:57:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:36.469+0000 7f0babfff700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:57:37.495 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:37.492+0000 7f5d93115000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 msgr2=0x7f0b98005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.172:3300/0 conn(0x7f0b98003760 0x7f0b98005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 msgr2=0x7f0b98008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b980062a0 0x7f0b98008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb0f8a700 1 -- 172.21.15.151:0/4199108043 reap_dead start 2023-09-30T01:57:38.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b98003ba0 0x7f0b98006720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac06dae0 con 0x7f0bac0ab5c0 2023-09-30T01:57:38.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98008860 con 0x7f0b98006c60 2023-09-30T01:57:38.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0ba9ffb700 1 -- 172.21.15.151:0/4199108043 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac06d800 con 0x7f0b98003ba0 2023-09-30T01:57:38.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> v2:172.21.15.151:3300/0 conn(0x7f0b98003ba0 0x7f0b98006720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v1:172.21.15.172:6789/0 conn(0x7f0b98006c60 legacy=0x7f0b98007100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 >> v1:172.21.15.151:6789/0 conn(0x7f0bac0ab5c0 legacy=0x7f0b9800b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bab7fe700 1 -- 172.21.15.151:0/4199108043 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98008ac0 con 0x7f0b98003ba0 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0b98006720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0006f30 tx=0x7f0ba0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564b02d3d422ac65 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0baaffd700 1 -- 172.21.15.151:0/4199108043 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba001d070 con 0x7f0b98003ba0 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0baaffd700 1 -- 172.21.15.151:0/4199108043 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba0018af0 con 0x7f0b98003ba0 2023-09-30T01:57:38.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0baaffd700 1 -- 172.21.15.151:0/4199108043 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0022420 con 0x7f0b98003ba0 2023-09-30T01:57:38.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 -- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 msgr2=0x7f0b98006720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0b98006720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0006f30 tx=0x7f0ba0008040 comp rx=0 tx=0).stop 2023-09-30T01:57:38.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 -- 172.21.15.151:0/4199108043 shutdown_connections 2023-09-30T01:57:38.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/4199108043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0b98006720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 -- 172.21.15.151:0/4199108043 >> 172.21.15.151:0/4199108043 conn(0x7f0bac070510 msgr2=0x7f0bac070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 -- 172.21.15.151:0/4199108043 shutdown_connections 2023-09-30T01:57:38.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.061+0000 7f0bb29ed700 1 -- 172.21.15.151:0/4199108043 wait complete. 2023-09-30T01:57:38.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 Processor -- start 2023-09-30T01:57:38.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 -- start start 2023-09-30T01:57:38.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b98006c60 0x7f0bac150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1510c0 con 0x7f0b98003ba0 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1363d0 con 0x7f0b98006c60 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.065+0000 7f0bb29ed700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac151370 con 0x7f0bac0ab5c0 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45448/0 (socket says 172.21.15.151:45448) 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 -- 172.21.15.151:0/3873825726 learned_addr learned my addr 172.21.15.151:0/3873825726 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb0f8a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb0f8a700 1 -- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 msgr2=0x7f0bac0a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb0f8a700 1 -- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 msgr2=0x7f0bac0a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb0f8a700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb0f8a700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bab7fe700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b98006c60 0x7f0bac150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 -- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 msgr2=0x7f0bac0a8850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 -- 172.21.15.151:0/3873825726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b98006c60 msgr2=0x7f0bac150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b98006c60 0x7f0bac150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 -- 172.21.15.151:0/3873825726 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba000e040 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0babfff700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4011cc0 tx=0x7f0ba4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4010af0 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4010c50 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac0afee0 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba401a440 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac151a20 con 0x7f0b98003ba0 2023-09-30T01:57:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0ba401a870 con 0x7f0b98003ba0 2023-09-30T01:57:38.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0ba4023990 con 0x7f0b98003ba0 2023-09-30T01:57:38.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.069+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac0670a0 con 0x7f0b98003ba0 2023-09-30T01:57:38.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.073+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0ba40396d0 con 0x7f0b98003ba0 2023-09-30T01:57:38.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.141+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0bac06c9a0 con 0x7f0b98003ba0 2023-09-30T01:57:38.153 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0b93fff700 1 -- 172.21.15.151:0/3873825726 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0ba4028070 con 0x7f0b98003ba0 2023-09-30T01:57:38.154 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 msgr2=0x7f0bac136d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.154 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4011cc0 tx=0x7f0ba4009040 comp rx=0 tx=0).stop 2023-09-30T01:57:38.154 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 shutdown_connections 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bac0ab5c0 0x7f0bac0a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b98006c60 0x7f0bac150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 --2- 172.21.15.151:0/3873825726 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b98003ba0 0x7f0bac136d50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 >> 172.21.15.151:0/3873825726 conn(0x7f0bac070510 msgr2=0x7f0bac139230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 shutdown_connections 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:38.149+0000 7f0bb29ed700 1 -- 172.21.15.151:0/3873825726 wait complete. 2023-09-30T01:57:38.155 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2023-09-30T01:57:38.157 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.156+0000 7f5d93115000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:57:38.168 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T01:57:38.168 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:38.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T01:57:38.183 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:38.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T01:57:38.195 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:38.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T01:57:38.206 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-09-30T01:57:38.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T01:57:38.213 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.212+0000 7f5d93115000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:57:38.262 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:38.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T01:57:38.278 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:38.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T01:57:38.281 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:38.277+0000 7fdee75a4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:57:38.329 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.328+0000 7f5d93115000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:57:38.336 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:38.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T01:57:38.345 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-09-30T01:57:38.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T01:57:38.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 8e644ce0-bab6-470f-8040-4b28674cfa75 0 2023-09-30T01:57:38.466 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 Processor -- start 2023-09-30T01:57:38.466 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- start start 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806a300 0x7fbb0806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb080acec0 0x7fbb0806e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0806a6d0 con 0x7fbb0806e540 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0806dcc0 con 0x7fbb080acec0 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0806c310 con 0x7fbb0806a300 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb080acec0 0x7fbb0806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.468 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0e9d8700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbb0806e540 0x7fbb0813d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44152/0 (socket says 172.21.15.172:44152) 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbb080acec0 0x7fbb0806e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38502/0 (socket says 172.21.15.172:38502) 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0e9d8700 1 -- 172.21.15.172:0/2720131521 learned_addr learned my addr 172.21.15.172:0/2720131521 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0e1d7700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806a300 0x7fbb0806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 -- 172.21.15.172:0/2720131521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806a300 msgr2=0x7fbb0806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806a300 0x7fbb0806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 -- 172.21.15.172:0/2720131521 >> v1:172.21.15.172:6789/0 conn(0x7fbb0806e540 legacy=0x7fbb0813d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d1d5700 1 -- 172.21.15.172:0/2720131521 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267640004 0 0) 0x7fbb0806a6d0 con 0x7fbb0806e540 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 -- 172.21.15.172:0/2720131521 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08137a10 con 0x7fbb080acec0 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d9d6700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb0806e000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80098e0 tx=0x7fbaf80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6de161be982c7585 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d1d5700 1 -- 172.21.15.172:0/2720131521 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8009e80 con 0x7fbb080acec0 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d1d5700 1 -- 172.21.15.172:0/2720131521 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf800b860 con 0x7fbb080acec0 2023-09-30T01:57:38.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0d1d5700 1 -- 172.21.15.172:0/2720131521 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf800bbe0 con 0x7fbb080acec0 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/2720131521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 msgr2=0x7fbb0806e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb0806e000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80098e0 tx=0x7fbaf80091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/2720131521 shutdown_connections 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb0806e000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/2720131521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806a300 0x7fbb0806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/2720131521 >> 172.21.15.172:0/2720131521 conn(0x7fbb080b0a60 msgr2=0x7fbb080b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.464+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/2720131521 shutdown_connections 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/2720131521 wait complete. 2023-09-30T01:57:38.473 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 Processor -- start 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- start start 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb0806a300 0x7fbb0813a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806e540 0x7fbb080a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080a6420 con 0x7fbb0806a300 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080a8090 con 0x7fbb080acec0 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080a6170 con 0x7fbb0806e540 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38504/0 (socket says 172.21.15.172:38504) 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 -- 172.21.15.172:0/1775668645 learned_addr learned my addr 172.21.15.172:0/1775668645 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:38.474 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e1d7700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb0806a300 0x7fbb0813a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0d9d6700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806e540 0x7fbb080a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 -- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806e540 msgr2=0x7fbb080a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806e540 0x7fbb080a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 -- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb0806a300 msgr2=0x7fbb0813a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb0806a300 0x7fbb0813a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 -- 172.21.15.172:0/1775668645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf8009420 con 0x7fbb080acec0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0e9d8700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000cc90 tx=0x7fbb00001a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000a8b0 con 0x7fbb080acec0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0000aa10 con 0x7fbb080acec0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0813aae0 con 0x7fbb080acec0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000add0 con 0x7fbb080acec0 2023-09-30T01:57:38.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0814bef0 con 0x7fbb080acec0 2023-09-30T01:57:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbb00007300 con 0x7fbb080acec0 2023-09-30T01:57:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbb0001b750 con 0x7fbb080acec0 2023-09-30T01:57:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.468+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb080008d0 con 0x7fbb080acec0 2023-09-30T01:57:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.472+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbb0814bef0 con 0x7fbb080acec0 2023-09-30T01:57:38.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.548+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8e644ce0-bab6-470f-8040-4b28674cfa75", "id": 0} v 0) v1 -- 0x7fbb0806bf50 con 0x7fbb080acec0 2023-09-30T01:57:38.555 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.552+0000 7f5d93115000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:57:38.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.556+0000 7fbafe7fc700 1 -- 172.21.15.172:0/1775668645 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e644ce0-bab6-470f-8040-4b28674cfa75", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbb00007790 con 0x7fbb080acec0 2023-09-30T01:57:38.560 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-09-30T01:57:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 msgr2=0x7fbb080a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000cc90 tx=0x7fbb00001a80 comp rx=0 tx=0).stop 2023-09-30T01:57:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 shutdown_connections 2023-09-30T01:57:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbb080acec0 0x7fbb080a6960 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb0806e540 0x7fbb080a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 --2- 172.21.15.172:0/1775668645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb0806a300 0x7fbb0813a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 >> 172.21.15.172:0/1775668645 conn(0x7fbb080b0a60 msgr2=0x7fbb0813de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 shutdown_connections 2023-09-30T01:57:38.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.560+0000 7fbb0f1d9700 1 -- 172.21.15.172:0/1775668645 wait complete. 2023-09-30T01:57:38.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new cff33849-65d5-4e21-a3f2-ac9f2aed071a 1 2023-09-30T01:57:38.606 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.604+0000 7f5d93115000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:57:38.672 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 Processor -- start 2023-09-30T01:57:38.672 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 -- start start 2023-09-30T01:57:38.675 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f80500acf60 0x7f805006e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.675 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805006e6d0 con 0x7f8050136c90 2023-09-30T01:57:38.675 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805006a980 con 0x7f805006a5b0 2023-09-30T01:57:38.675 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f8057b2d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805006ccb0 con 0x7f80500acf60 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.668+0000 7f80558c9700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f805006a5b0 0x7f805006c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44168/0 (socket says 172.21.15.172:44168) 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80558c9700 1 -- 172.21.15.172:0/2032492881 learned_addr learned my addr 172.21.15.172:0/2032492881 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- 172.21.15.172:0/2032492881 >> v2:172.21.15.151:3300/0 conn(0x7f80500acf60 0x7f805006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096791325 0 0) 0x7f805006a980 con 0x7f805006a5b0 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c003570 con 0x7f805006a5b0 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590879264 0 0) 0x7f803c003570 con 0x7f805006a5b0 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 >> v2:172.21.15.151:3300/0 conn(0x7f80500acf60 msgr2=0x7f805006e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 --2- 172.21.15.172:0/2032492881 >> v2:172.21.15.151:3300/0 conn(0x7f80500acf60 0x7f805006e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 >> v1:172.21.15.151:6789/0 conn(0x7f8050136c90 legacy=0x7f8050137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805006da90 con 0x7f805006a5b0 2023-09-30T01:57:38.676 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f804c001470 con 0x7f805006a5b0 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 >> v1:172.21.15.172:6789/0 conn(0x7f805006a5b0 legacy=0x7f805006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c003a60 0x7f803c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f803c0051c0 0x7f803c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805006ccb0 con 0x7f803c0051c0 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c00c0d0 con 0x7f803c004810 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c00c290 con 0x7f803c003a60 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80558c9700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c003a60 0x7f803c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 -- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c003a60 msgr2=0x7f803c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c003a60 0x7f803c003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.677 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 -- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f803c0051c0 msgr2=0x7f803c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f803c0051c0 0x7f803c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 -- 172.21.15.172:0/2032492881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c003570 con 0x7f803c004810 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80560ca700 1 -- 172.21.15.172:0/2032492881 reap_dead start 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 0x7f803c004c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80500ad370 tx=0x7f80400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5831a96ee83ab9d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000e710 con 0x7f803c004810 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804000e870 con 0x7f803c004810 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80548c7700 1 -- 172.21.15.172:0/2032492881 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000ec30 con 0x7f803c004810 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 msgr2=0x7f803c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 0x7f803c004c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80500ad370 tx=0x7f80400091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- 172.21.15.172:0/2032492881 shutdown_connections 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2032492881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.678 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- 172.21.15.172:0/2032492881 >> 172.21.15.172:0/2032492881 conn(0x7f80500b0b10 msgr2=0x7f80500b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.679 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- 172.21.15.172:0/2032492881 shutdown_connections 2023-09-30T01:57:38.679 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- 172.21.15.172:0/2032492881 wait complete. 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 Processor -- start 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- start start 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c004810 0x7f805013b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f805014df60 0x7f805013db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805014dad0 con 0x7f805014df60 2023-09-30T01:57:38.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805013d680 con 0x7f805013d210 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f8057b2d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805014d780 con 0x7f803c004810 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.672+0000 7f80550c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38522/0 (socket says 172.21.15.172:38522) 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 -- 172.21.15.172:0/2024214720 learned_addr learned my addr 172.21.15.172:0/2024214720 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80560ca700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f805014df60 0x7f805013db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80558c9700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c004810 0x7f805013b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 -- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c004810 msgr2=0x7f805013b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c004810 0x7f805013b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 -- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f805014df60 msgr2=0x7f805013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f805014df60 0x7f805013db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 -- 172.21.15.172:0/2024214720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805013ca60 con 0x7f805013d210 2023-09-30T01:57:38.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f80550c8700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c260 tx=0x7f804001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8040022070 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804000e9e0 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805013abf0 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000e710 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805014e570 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f804001c8e0 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f8040030840 con 0x7f805013d210 2023-09-30T01:57:38.683 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.676+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805014dc40 con 0x7f805013d210 2023-09-30T01:57:38.684 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.680+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8040035060 con 0x7f805013d210 2023-09-30T01:57:38.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.756+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cff33849-65d5-4e21-a3f2-ac9f2aed071a", "id": 1} v 0) v1 -- 0x7f80500670a0 con 0x7f805013d210 2023-09-30T01:57:38.768 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.764+0000 7f8045ffb700 1 -- 172.21.15.172:0/2024214720 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cff33849-65d5-4e21-a3f2-ac9f2aed071a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f804002b070 con 0x7f805013d210 2023-09-30T01:57:38.768 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-09-30T01:57:38.769 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 msgr2=0x7f805013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.769 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c260 tx=0x7f804001a000 comp rx=0 tx=0).stop 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 shutdown_connections 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f805014df60 0x7f805013db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f805013d210 0x7f805013a6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 --2- 172.21.15.172:0/2024214720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f803c004810 0x7f805013b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 >> 172.21.15.172:0/2024214720 conn(0x7f80500b0b10 msgr2=0x7f80500ab380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 shutdown_connections 2023-09-30T01:57:38.770 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.768+0000 7f8057b2d700 1 -- 172.21.15.172:0/2024214720 wait complete. 2023-09-30T01:57:38.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f 2 2023-09-30T01:57:38.784 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.780+0000 7f5d93115000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:57:38.843 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.840+0000 7f5d93115000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:57:38.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 Processor -- start 2023-09-30T01:57:38.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 -- start start 2023-09-30T01:57:38.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa813e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3aa80ab0f0 0x7f3aa813ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa80ab610 con 0x7f3aa813f130 2023-09-30T01:57:38.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa80ac160 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aaed89700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8070b10 con 0x7f3aa8070740 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3aa80ab0f0 0x7f3aa813ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aae588700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3aa813f130 0x7f3aa806d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44172/0 (socket says 172.21.15.172:44172) 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aae588700 1 -- 172.21.15.172:0/292576893 learned_addr learned my addr 172.21.15.172:0/292576893 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aadd87700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa813e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 -- 172.21.15.172:0/292576893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 msgr2=0x7f3aa813e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa813e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 -- 172.21.15.172:0/292576893 >> v1:172.21.15.172:6789/0 conn(0x7f3aa813f130 legacy=0x7f3aa806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 -- 172.21.15.172:0/292576893 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8068160 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aacd85700 1 -- 172.21.15.172:0/292576893 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011243472 0 0) 0x7f3aa80ab610 con 0x7f3aa813f130 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aad586700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa813ebf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e53478661ace123 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aacd85700 1 -- 172.21.15.172:0/292576893 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800b640 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aacd85700 1 -- 172.21.15.172:0/292576893 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9800b7a0 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.884+0000 7f3aacd85700 1 -- 172.21.15.172:0/292576893 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800bb40 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.892 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/292576893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80ab0f0 msgr2=0x7f3aa813ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa813ebf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a980091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/292576893 shutdown_connections 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa813ebf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- 172.21.15.172:0/292576893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa813e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/292576893 >> 172.21.15.172:0/292576893 conn(0x7f3aa80b0400 msgr2=0x7f3aa80b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/292576893 shutdown_connections 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/292576893 wait complete. 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 Processor -- start 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- start start 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa80aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa80aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8136cb0 con 0x7f3aa80ab0f0 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8136f30 con 0x7f3aa813f130 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8135c70 con 0x7f3aa8070740 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38544/0 (socket says 172.21.15.172:38544) 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 -- 172.21.15.172:0/3702390987 learned_addr learned my addr 172.21.15.172:0/3702390987 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aadd87700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa80aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aad586700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa80aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 -- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 msgr2=0x7f3aa80aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa80aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 -- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80ab0f0 msgr2=0x7f3aa80aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa80aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 -- 172.21.15.172:0/3702390987 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98009420 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aae588700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000cc10 tx=0x7f3aa00019c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000a8b0 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa000aa30 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000adf0 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa806d4d0 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa8137b70 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3aa0007300 con 0x7f3aa813f130 2023-09-30T01:57:38.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.888+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f3aa0012400 con 0x7f3aa813f130 2023-09-30T01:57:38.896 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.892+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa81370f0 con 0x7f3aa813f130 2023-09-30T01:57:38.896 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.892+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3aa00075b0 con 0x7f3aa813f130 2023-09-30T01:57:38.903 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:38.900+0000 7f5d93115000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:57:38.903 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:38.901+0000 7fdee75a4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:57:38.960 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:38.957+0000 7fdee75a4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:57:38.971 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.968+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f", "id": 2} v 0) v1 -- 0x7f3aa80008d0 con 0x7f3aa813f130 2023-09-30T01:57:38.980 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.976+0000 7f3a9dffb700 1 -- 172.21.15.172:0/3702390987 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3aa0007850 con 0x7f3aa813f130 2023-09-30T01:57:38.980 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-09-30T01:57:38.981 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 msgr2=0x7f3aa8137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:38.981 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000cc10 tx=0x7f3aa00019c0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.981 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 shutdown_connections 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa813f130 0x7f3aa8137410 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80ab0f0 0x7f3aa80aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 --2- 172.21.15.172:0/3702390987 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa8070740 0x7f3aa80aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 >> 172.21.15.172:0/3702390987 conn(0x7f3aa80b0400 msgr2=0x7f3aa80ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 shutdown_connections 2023-09-30T01:57:38.982 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:38.980+0000 7f3aaed89700 1 -- 172.21.15.172:0/3702390987 wait complete. 2023-09-30T01:57:38.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new fd981572-4e9d-4d6c-8c40-5e7bd24eadc0 3 2023-09-30T01:57:39.071 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.065+0000 7fdee75a4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:57:39.140 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.136+0000 7f0080def700 1 Processor -- start 2023-09-30T01:57:39.140 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.136+0000 7f0080def700 1 -- start start 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f007c070510 0x7f007c1370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c06f290 con 0x7f007c0ac300 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c06e180 con 0x7f007c0708e0 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c06e3f0 con 0x7f007c070510 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f007c070510 0x7f007c1370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f007c070510 0x7f007c1370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38558/0 (socket says 172.21.15.172:38558) 2023-09-30T01:57:39.143 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 -- 172.21.15.172:0/2759355154 learned_addr learned my addr 172.21.15.172:0/2759355154 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.143 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 -- 172.21.15.172:0/2759355154 >> v1:172.21.15.172:6789/0 conn(0x7f007c0708e0 legacy=0x7f007c137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.143 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 -- 172.21.15.172:0/2759355154 >> v1:172.21.15.151:6789/0 conn(0x7f007c0ac300 legacy=0x7f007c13a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.143 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 -- 172.21.15.172:0/2759355154 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c06ede0 con 0x7f007c070510 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007b7fe700 1 --2- 172.21.15.172:0/2759355154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c070510 0x7f007c1370d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f006c0099b0 tx=0x7f006c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58121642b6a1a34 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007affd700 1 -- 172.21.15.172:0/2759355154 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c00b860 con 0x7f007c070510 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007affd700 1 -- 172.21.15.172:0/2759355154 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c00b9c0 con 0x7f007c070510 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f007affd700 1 -- 172.21.15.172:0/2759355154 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c00bd60 con 0x7f007c070510 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- 172.21.15.172:0/2759355154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c070510 msgr2=0x7f007c1370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 --2- 172.21.15.172:0/2759355154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c070510 0x7f007c1370d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f006c0099b0 tx=0x7f006c009250 comp rx=0 tx=0).stop 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- 172.21.15.172:0/2759355154 shutdown_connections 2023-09-30T01:57:39.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 --2- 172.21.15.172:0/2759355154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c070510 0x7f007c1370d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- 172.21.15.172:0/2759355154 >> 172.21.15.172:0/2759355154 conn(0x7f007c0b0400 msgr2=0x7f007c0b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- 172.21.15.172:0/2759355154 shutdown_connections 2023-09-30T01:57:39.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.140+0000 7f0080def700 1 -- 172.21.15.172:0/2759355154 wait complete. 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 Processor -- start 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- start start 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f007c070510 0x7f007c0a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f007c0ac300 0x7f007c1315b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0a9630 con 0x7f007c070510 2023-09-30T01:57:39.147 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0a9350 con 0x7f007c0708e0 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c133430 con 0x7f007c0ac300 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38572/0 (socket says 172.21.15.172:38572) 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 -- 172.21.15.172:0/2315546614 learned_addr learned my addr 172.21.15.172:0/2315546614 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f007b7fe700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f007c070510 0x7f007c0a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f007bfff700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f007c0ac300 0x7f007c1315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 -- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f007c0ac300 msgr2=0x7f007c1315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f007c0ac300 0x7f007c1315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 -- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f007c070510 msgr2=0x7f007c0a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f007c070510 0x7f007c0a8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 -- 172.21.15.172:0/2315546614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0094f0 con 0x7f007c0708e0 2023-09-30T01:57:39.148 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0073fff700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f006400c820 tx=0x7f0064007f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064011590 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064011710 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c1336a0 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064011ad0 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c131d10 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f006401c6c0 con 0x7f007c0708e0 2023-09-30T01:57:39.149 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c0a9770 con 0x7f007c0708e0 2023-09-30T01:57:39.150 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.144+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f0064011e40 con 0x7f007c0708e0 2023-09-30T01:57:39.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.148+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0064017070 con 0x7f007c0708e0 2023-09-30T01:57:39.228 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.224+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fd981572-4e9d-4d6c-8c40-5e7bd24eadc0", "id": 3} v 0) v1 -- 0x7f007c0008d0 con 0x7f007c0708e0 2023-09-30T01:57:39.236 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.232+0000 7f00737fe700 1 -- 172.21.15.172:0/2315546614 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd981572-4e9d-4d6c-8c40-5e7bd24eadc0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f006401b470 con 0x7f007c0708e0 2023-09-30T01:57:39.236 INFO:teuthology.orchestra.run.smithi172.stdout:3 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 msgr2=0x7f007c13d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f006400c820 tx=0x7f0064007f50 comp rx=0 tx=0).stop 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 shutdown_connections 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f007c0ac300 0x7f007c1315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f007c0708e0 0x7f007c13d940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 --2- 172.21.15.172:0/2315546614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f007c070510 0x7f007c0a8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 >> 172.21.15.172:0/2315546614 conn(0x7f007c0b0400 msgr2=0x7f007c06be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.239 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 shutdown_connections 2023-09-30T01:57:39.239 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.236+0000 7f0080def700 1 -- 172.21.15.172:0/2315546614 wait complete. 2023-09-30T01:57:39.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 0ac4613b-f5e3-475e-a5d5-71175f061a97 4 2023-09-30T01:57:39.289 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.285+0000 7fdee75a4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:57:39.306 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.304+0000 7f5d93115000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:57:39.339 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.333+0000 7fdee75a4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 Processor -- start 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 -- start start 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4bbc070740 0x7f4bbc0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc13d380 con 0x7f4bbc06f220 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc070b10 con 0x7f4bbc13d810 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.348+0000 7f4bc19b5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0ac3d0 con 0x7f4bbc070740 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb3fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4bbc13d810 0x7f4bbc06eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44186/0 (socket says 172.21.15.172:44186) 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb3fff700 1 -- 172.21.15.172:0/522748873 learned_addr learned my addr 172.21.15.172:0/522748873 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc09b3700 1 --2- 172.21.15.172:0/522748873 >> v2:172.21.15.151:3300/0 conn(0x7f4bbc070740 0x7f4bbc0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610839861 0 0) 0x7f4bbc070b10 con 0x7f4bbc13d810 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4003570 con 0x7f4bbc13d810 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573224410 0 0) 0x7f4ba4003570 con 0x7f4bbc13d810 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 >> v2:172.21.15.151:3300/0 conn(0x7f4bbc070740 msgr2=0x7f4bbc0aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 --2- 172.21.15.172:0/522748873 >> v2:172.21.15.151:3300/0 conn(0x7f4bbc070740 0x7f4bbc0aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 >> v1:172.21.15.151:6789/0 conn(0x7f4bbc06f220 legacy=0x7f4bbc147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0aa4e0 con 0x7f4bbc13d810 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f4bac001470 con 0x7f4bbc13d810 2023-09-30T01:57:39.357 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 >> v1:172.21.15.172:6789/0 conn(0x7f4bbc13d810 legacy=0x7f4bbc06eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ba4003a60 0x7f4ba4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba4004810 0x7f4ba4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 0x7f4ba400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0ac3d0 con 0x7f4ba4004810 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400c160 con 0x7f4ba40051c0 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400c320 con 0x7f4ba4003a60 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 0x7f4ba400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc09b3700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ba4003a60 0x7f4ba4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb3fff700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba4004810 0x7f4ba4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 -- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ba4003a60 msgr2=0x7f4ba4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.358 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ba4003a60 0x7f4ba4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 -- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba4004810 msgr2=0x7f4ba4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba4004810 0x7f4ba4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 -- 172.21.15.172:0/522748873 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4003570 con 0x7f4ba40051c0 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc09b3700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ba4003a60 0x7f4ba4003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 -- 172.21.15.172:0/522748873 reap_dead start 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc11b4700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 0x7f4ba400ba60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4001fc0 tx=0x7f4bb4001640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a828f59c9188ce6b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4001950 con 0x7f4ba40051c0 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb400c8b0 con 0x7f4ba40051c0 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bb37fe700 1 -- 172.21.15.172:0/522748873 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400cc90 con 0x7f4ba40051c0 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 -- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 msgr2=0x7f4ba400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.359 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 0x7f4ba400ba60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4001fc0 tx=0x7f4bb4001640 comp rx=0 tx=0).stop 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 -- 172.21.15.172:0/522748873 shutdown_connections 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/522748873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ba40051c0 0x7f4ba400ba60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 -- 172.21.15.172:0/522748873 >> 172.21.15.172:0/522748873 conn(0x7f4bbc0b0400 msgr2=0x7f4bbc0b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 -- 172.21.15.172:0/522748873 shutdown_connections 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 -- 172.21.15.172:0/522748873 wait complete. 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.352+0000 7f4bc19b5700 1 Processor -- start 2023-09-30T01:57:39.360 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- start start 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba40051c0 0x7f4bbc06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4bbc14ff10 0x7f4bbc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc137800 con 0x7f4ba40051c0 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc06cc90 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc1372a0 con 0x7f4bbc14ff10 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38586/0 (socket says 172.21.15.172:38586) 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 -- 172.21.15.172:0/3437826095 learned_addr learned my addr 172.21.15.172:0/3437826095 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc09b3700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba40051c0 0x7f4bbc06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc11b4700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4bbc14ff10 0x7f4bbc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 -- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4bbc14ff10 msgr2=0x7f4bbc06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4bbc14ff10 0x7f4bbc06b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 -- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba40051c0 msgr2=0x7f4bbc06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba40051c0 0x7f4bbc06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 -- 172.21.15.172:0/3437826095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc150380 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb3fff700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00dbb0 tx=0x7f4bac009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac007e90 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac007360 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc1505d0 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc06beb0 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac004830 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4bac00b600 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f4bac01e950 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.356+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc0008d0 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.360+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4bac023060 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.416 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.412+0000 7f5d93115000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:57:39.438 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.436+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0ac4613b-f5e3-475e-a5d5-71175f061a97", "id": 4} v 0) v1 -- 0x7f4bbc0af960 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.447 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bb0ff9700 1 -- 172.21.15.172:0/3437826095 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ac4613b-f5e3-475e-a5d5-71175f061a97", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4bac019070 con 0x7f4bbc06d4c0 2023-09-30T01:57:39.447 INFO:teuthology.orchestra.run.smithi172.stdout:4 2023-09-30T01:57:39.448 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 msgr2=0x7f4bbc136b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00dbb0 tx=0x7f4bac009040 comp rx=0 tx=0).stop 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 shutdown_connections 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4bbc14ff10 0x7f4bbc06b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4bbc06d4c0 0x7f4bbc136b40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 --2- 172.21.15.172:0/3437826095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ba40051c0 0x7f4bbc06def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 >> 172.21.15.172:0/3437826095 conn(0x7f4bbc0b0400 msgr2=0x7f4bbc13b670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 shutdown_connections 2023-09-30T01:57:39.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.444+0000 7f4bc19b5700 1 -- 172.21.15.172:0/3437826095 wait complete. 2023-09-30T01:57:39.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new d833c90c-9efc-4367-8b92-b01219b686e3 5 2023-09-30T01:57:39.511 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.505+0000 7fdee75a4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:57:39.565 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 Processor -- start 2023-09-30T01:57:39.565 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 -- start start 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f21880acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880ae9f0 con 0x7f218806eb60 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218806d060 con 0x7f218806ddf0 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21880ad440 con 0x7f21880ae130 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f2186d9d700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f218806ddf0 0x7f218813eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:44200/0 (socket says 172.21.15.172:44200) 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f2186d9d700 1 -- 172.21.15.172:0/21801185 learned_addr learned my addr 172.21.15.172:0/21801185 (peer_addr_for_me v1:172.21.15.172:0/0) 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f21880acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446635426 0 0) 0x7f218806d060 con 0x7f218806ddf0 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2174003570 con 0x7f218806ddf0 2023-09-30T01:57:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585765518 0 0) 0x7f21880ae9f0 con 0x7f218806eb60 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f218806d060 con 0x7f218806eb60 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.1 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852029424 0 0) 0x7f2174003570 con 0x7f218806ddf0 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 msgr2=0x7f21880acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f21880acde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 >> v1:172.21.15.151:6789/0 conn(0x7f218806eb60 legacy=0x7f218806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21880ae500 con 0x7f218806ddf0 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.1 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f21800017f0 con 0x7f218806ddf0 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 >> v1:172.21.15.172:6789/0 conn(0x7f218806ddf0 legacy=0x7f218813eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f2174003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21740041a0 0x7f2174004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.569 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2174004d50 0x7f21740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880ad440 con 0x7f21740041a0 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217400bc70 con 0x7f2174004d50 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217400be30 con 0x7f21880ae130 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f2174003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f2186d9d700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2174004d50 0x7f21740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f2185d9b700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21740041a0 0x7f2174004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 -- 172.21.15.172:0/21801185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2174004d50 msgr2=0x7f21740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2174004d50 0x7f21740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 -- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21740041a0 msgr2=0x7f2174004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21740041a0 0x7f2174004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 -- 172.21.15.172:0/21801185 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2174003570 con 0x7f21880ae130 2023-09-30T01:57:39.570 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218659c700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f2174003bf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2178006f00 tx=0x7f21780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37bc9fe9a2853f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780095b0 con 0x7f21880ae130 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217800e710 con 0x7f21880ae130 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.564+0000 7f218559a700 1 -- 172.21.15.172:0/21801185 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217800eab0 con 0x7f21880ae130 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 msgr2=0x7f2174003bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 --2- 172.21.15.172:0/21801185 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21880ae130 0x7f2174003bf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2178006f00 tx=0x7f21780091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2186d9d700 1 -- 172.21.15.172:0/21801185 reap_dead start 2023-09-30T01:57:39.574 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- 172.21.15.172:0/21801185 shutdown_connections 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- 172.21.15.172:0/21801185 >> 172.21.15.172:0/21801185 conn(0x7f2188070510 msgr2=0x7f21880708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- 172.21.15.172:0/21801185 shutdown_connections 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- 172.21.15.172:0/21801185 wait complete. 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 Processor -- start 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- start start 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 0x7f2188137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2188138120 0x7f21880aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880a9590 con 0x7f21880a8250 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880a9870 con 0x7f21880ae130 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21880a86e0 con 0x7f2188138120 2023-09-30T01:57:39.575 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38616/0 (socket says 172.21.15.172:38616) 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218659c700 1 -- 172.21.15.172:0/1370377852 learned_addr learned my addr 172.21.15.172:0/1370377852 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2186d9d700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2188138120 0x7f21880aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 0x7f2188137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 -- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2188138120 msgr2=0x7f21880aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2188138120 0x7f21880aa050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 -- 172.21.15.172:0/1370377852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 msgr2=0x7f21880a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f2185d9b700 1 -- 172.21.15.172:0/1370377852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21880a6610 con 0x7f21880a8250 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.568+0000 7f218659c700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f2185d9b700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 0x7f2188137be0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2170002a80 tx=0x7f2170007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217000f8d0 con 0x7f21880a8250 2023-09-30T01:57:39.577 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217000fa50 con 0x7f21880a8250 2023-09-30T01:57:39.577 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2188138590 con 0x7f21880a8250 2023-09-30T01:57:39.577 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217000fe10 con 0x7f21880a8250 2023-09-30T01:57:39.577 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21880aa730 con 0x7f21880a8250 2023-09-30T01:57:39.578 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2170013880 con 0x7f21880a8250 2023-09-30T01:57:39.578 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f2170013b30 con 0x7f21880a8250 2023-09-30T01:57:39.578 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.569+0000 7fdee75a4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:57:39.579 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.572+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21880008d0 con 0x7f21880a8250 2023-09-30T01:57:39.581 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.580+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f217003b6d0 con 0x7f21880a8250 2023-09-30T01:57:39.635 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:39.629+0000 7fdee75a4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:57:39.658 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.656+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d833c90c-9efc-4367-8b92-b01219b686e3", "id": 5} v 0) v1 -- 0x7f218813ca10 con 0x7f21880a8250 2023-09-30T01:57:39.665 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f217e7fc700 1 -- 172.21.15.172:0/1370377852 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d833c90c-9efc-4367-8b92-b01219b686e3", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f217002a070 con 0x7f21880a8250 2023-09-30T01:57:39.665 INFO:teuthology.orchestra.run.smithi172.stdout:5 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 msgr2=0x7f2188137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 0x7f2188137be0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2170002a80 tx=0x7f2170007d30 comp rx=0 tx=0).stop 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 shutdown_connections 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2188138120 0x7f21880aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21880a8250 0x7f2188137be0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 --2- 172.21.15.172:0/1370377852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21880ae130 0x7f21880a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 >> 172.21.15.172:0/1370377852 conn(0x7f2188070510 msgr2=0x7f2188062d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 shutdown_connections 2023-09-30T01:57:39.667 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.664+0000 7f218759e700 1 -- 172.21.15.172:0/1370377852 wait complete. 2023-09-30T01:57:39.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new b54db1f9-34f1-4ef9-96a1-fde5a9555808 6 2023-09-30T01:57:39.714 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.712+0000 7f5d93115000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:57:39.781 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.776+0000 7f5cc6a35700 1 Processor -- start 2023-09-30T01:57:39.781 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.776+0000 7f5cc6a35700 1 -- start start 2023-09-30T01:57:39.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5cc006e1d0 0x7f5cc00abc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc00acc30 con 0x7f5cc00ab250 2023-09-30T01:57:39.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc0138d30 con 0x7f5cc006e1d0 2023-09-30T01:57:39.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00acf10 con 0x7f5cc006e5a0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5cc006e1d0 0x7f5cc00abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5cc006e1d0 0x7f5cc00abc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38630/0 (socket says 172.21.15.172:38630) 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 -- 172.21.15.172:0/2614391724 learned_addr learned my addr 172.21.15.172:0/2614391724 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5232700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 -- 172.21.15.172:0/2614391724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 msgr2=0x7f5cc013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc013b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 -- 172.21.15.172:0/2614391724 >> v1:172.21.15.151:6789/0 conn(0x7f5cc00ab250 legacy=0x7f5cc00a9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 -- 172.21.15.172:0/2614391724 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc00aaef0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc5a33700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc00abc50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0099b0 tx=0x7f5cbc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89bbbf276318d0be server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc4a31700 1 -- 172.21.15.172:0/2614391724 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00b860 con 0x7f5cc006e1d0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc4a31700 1 -- 172.21.15.172:0/2614391724 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00b9c0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.784 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc4a31700 1 -- 172.21.15.172:0/2614391724 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00bd60 con 0x7f5cc006e1d0 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- 172.21.15.172:0/2614391724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 msgr2=0x7f5cc00abc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc00abc50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0099b0 tx=0x7f5cbc009250 comp rx=0 tx=0).stop 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- 172.21.15.172:0/2614391724 shutdown_connections 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/2614391724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc00abc50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- 172.21.15.172:0/2614391724 >> 172.21.15.172:0/2614391724 conn(0x7f5cc00b0a80 msgr2=0x7f5cc00b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- 172.21.15.172:0/2614391724 shutdown_connections 2023-09-30T01:57:39.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 -- 172.21.15.172:0/2614391724 wait complete. 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.780+0000 7f5cc6a35700 1 Processor -- start 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- start start 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc00a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc00ab250 0x7f5cc006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00a9450 con 0x7f5cc00ab250 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00a9230 con 0x7f5cc006e1d0 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc006bfc0 con 0x7f5cc006e5a0 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38638/0 (socket says 172.21.15.172:38638) 2023-09-30T01:57:39.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 -- 172.21.15.172:0/3485298436 learned_addr learned my addr 172.21.15.172:0/3485298436 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6234700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc00ab250 0x7f5cc006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5232700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc00a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 -- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 msgr2=0x7f5cc00a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc00a8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 -- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc00ab250 msgr2=0x7f5cc006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc00ab250 0x7f5cc006d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 -- 172.21.15.172:0/3485298436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0094f0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5232700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc00a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc5a33700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0068450 tx=0x7f5cbc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc024440 con 0x7f5cc006e1d0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc0245c0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc01381e0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc024980 con 0x7f5cc006e1d0 2023-09-30T01:57:39.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc006dbe0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5cbc01a6a0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f5cbc0239e0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.784+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc00008d0 con 0x7f5cc006e1d0 2023-09-30T01:57:39.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.788+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5cbc015020 con 0x7f5cc006e1d0 2023-09-30T01:57:39.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.864+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b54db1f9-34f1-4ef9-96a1-fde5a9555808", "id": 6} v 0) v1 -- 0x7f5cc00ab730 con 0x7f5cc006e1d0 2023-09-30T01:57:39.873 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cb5ffb700 1 -- 172.21.15.172:0/3485298436 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b54db1f9-34f1-4ef9-96a1-fde5a9555808", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5cbc01ab80 con 0x7f5cc006e1d0 2023-09-30T01:57:39.874 INFO:teuthology.orchestra.run.smithi172.stdout:6 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 msgr2=0x7f5cc006cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0068450 tx=0x7f5cbc017000 comp rx=0 tx=0).stop 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 shutdown_connections 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cc00ab250 0x7f5cc006d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cc006e5a0 0x7f5cc00a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 --2- 172.21.15.172:0/3485298436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5cc006e1d0 0x7f5cc006cc40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 >> 172.21.15.172:0/3485298436 conn(0x7f5cc00b0a80 msgr2=0x7f5cc0067ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 shutdown_connections 2023-09-30T01:57:39.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5cc6a35700 1 -- 172.21.15.172:0/3485298436 wait complete. 2023-09-30T01:57:39.876 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.872+0000 7f5d93115000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:57:39.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 24126a91-6681-47fb-87f8-490864058d02 7 2023-09-30T01:57:39.946 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.944+0000 7f5d93115000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:57:39.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 Processor -- start 2023-09-30T01:57:39.991 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 -- start start 2023-09-30T01:57:39.992 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13a80706b0 0x7f13a80ab380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f13a813ec30 0x7f13a813f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a806db50 con 0x7f13a813b380 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a80abbe0 con 0x7f13a813ec30 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13acb58700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a80ac6b0 con 0x7f13a80706b0 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13a659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13a80706b0 0x7f13a80ab380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13a659c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f13a80706b0 0x7f13a80ab380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38648/0 (socket says 172.21.15.172:38648) 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13a659c700 1 -- 172.21.15.172:0/3925902 learned_addr learned my addr 172.21.15.172:0/3925902 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.988+0000 7f13a5d9b700 1 --2- 172.21.15.172:0/3925902 >> v2:172.21.15.151:3300/0 conn(0x7f13a813ec30 0x7f13a813f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 -- 172.21.15.172:0/3925902 >> v2:172.21.15.151:3300/0 conn(0x7f13a813ec30 msgr2=0x7f13a813f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 --2- 172.21.15.172:0/3925902 >> v2:172.21.15.151:3300/0 conn(0x7f13a813ec30 0x7f13a813f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.993 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 -- 172.21.15.172:0/3925902 >> v1:172.21.15.151:6789/0 conn(0x7f13a813b380 legacy=0x7f13a813b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.994 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 -- 172.21.15.172:0/3925902 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80ac920 con 0x7f13a80706b0 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 --2- 172.21.15.172:0/3925902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80ab380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13980099b0 tx=0x7f1398009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977d1b85cf41edab server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a559a700 1 -- 172.21.15.172:0/3925902 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398009ea0 con 0x7f13a80706b0 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a559a700 1 -- 172.21.15.172:0/3925902 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139800b820 con 0x7f13a80706b0 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a559a700 1 -- 172.21.15.172:0/3925902 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139800bbc0 con 0x7f13a80706b0 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- 172.21.15.172:0/3925902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 msgr2=0x7f13a80ab380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- 172.21.15.172:0/3925902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80ab380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13980099b0 tx=0x7f1398009250 comp rx=0 tx=0).stop 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- 172.21.15.172:0/3925902 shutdown_connections 2023-09-30T01:57:39.995 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- 172.21.15.172:0/3925902 >> v2:172.21.15.151:3300/0 conn(0x7f13a813ec30 0x7f13a813f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- 172.21.15.172:0/3925902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80ab380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- 172.21.15.172:0/3925902 >> 172.21.15.172:0/3925902 conn(0x7f13a8067e00 msgr2=0x7f13a80b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- 172.21.15.172:0/3925902 shutdown_connections 2023-09-30T01:57:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- 172.21.15.172:0/3925902 wait complete. 2023-09-30T01:57:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 Processor -- start 2023-09-30T01:57:39.996 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f5d93115000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- start start 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 0x7f13a814b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a814b3b0 con 0x7f13a813b380 2023-09-30T01:57:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80a8c50 con 0x7f13a80706b0 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13acb58700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8118190 con 0x7f13a813ec30 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38652/0 (socket says 172.21.15.172:38652) 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.172:59406/0 (socket says 172.21.15.172:59406) 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 -- 172.21.15.172:0/1481786923 learned_addr learned my addr 172.21.15.172:0/1481786923 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a6d9d700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 0x7f13a814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 -- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 msgr2=0x7f13a814b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 0x7f13a814b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 -- 172.21.15.172:0/1481786923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 msgr2=0x7f13a80a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a5d9b700 1 -- 172.21.15.172:0/1481786923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13980094f0 con 0x7f13a813b380 2023-09-30T01:57:40.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a6d9d700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 0x7f13a814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:40.003 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.992+0000 7f13a659c700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13a5d9b700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e930 tx=0x7f139c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c013680 con 0x7f13a813b380 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c0137e0 con 0x7f13a813b380 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a80a6af0 con 0x7f13a813b380 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c013b80 con 0x7f13a813b380 2023-09-30T01:57:40.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a814bd80 con 0x7f13a813b380 2023-09-30T01:57:40.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f139c01c6d0 con 0x7f13a813b380 2023-09-30T01:57:40.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f139c012a40 con 0x7f13a813b380 2023-09-30T01:57:40.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:39.996+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a814b560 con 0x7f13a813b380 2023-09-30T01:57:40.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.000+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f13a814b560 con 0x7f13a813b380 2023-09-30T01:57:40.022 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.017+0000 7fdee75a4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:57:40.053 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.052+0000 7f5d93115000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:57:40.080 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.076+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "24126a91-6681-47fb-87f8-490864058d02", "id": 7} v 0) v1 -- 0x7f13a80670a0 con 0x7f13a813b380 2023-09-30T01:57:40.086 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13967fc700 1 -- 172.21.15.172:0/1481786923 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24126a91-6681-47fb-87f8-490864058d02", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f139c021060 con 0x7f13a813b380 2023-09-30T01:57:40.086 INFO:teuthology.orchestra.run.smithi172.stdout:7 2023-09-30T01:57:40.088 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 msgr2=0x7f13a80a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.088 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e930 tx=0x7f139c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 shutdown_connections 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13a813ec30 0x7f13a814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13a813b380 0x7f13a80a8710 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 --2- 172.21.15.172:0/1481786923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13a80706b0 0x7f13a80a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 >> 172.21.15.172:0/1481786923 conn(0x7f13a8067e00 msgr2=0x7f13a8139280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 shutdown_connections 2023-09-30T01:57:40.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-09-30T01:57:40.084+0000 7f13acb58700 1 -- 172.21.15.172:0/1481786923 wait complete. 2023-09-30T01:57:40.104 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T01:57:40.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T01:57:40.107 INFO:tasks.ceph.osd.0:Started 2023-09-30T01:57:40.107 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T01:57:40.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T01:57:40.109 INFO:tasks.ceph.osd.1:Started 2023-09-30T01:57:40.109 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T01:57:40.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T01:57:40.111 INFO:tasks.ceph.osd.2:Started 2023-09-30T01:57:40.112 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T01:57:40.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T01:57:40.114 INFO:tasks.ceph.osd.3:Started 2023-09-30T01:57:40.114 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T01:57:40.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T01:57:40.116 INFO:tasks.ceph.osd.4:Started 2023-09-30T01:57:40.116 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T01:57:40.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T01:57:40.118 INFO:tasks.ceph.osd.5:Started 2023-09-30T01:57:40.118 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T01:57:40.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T01:57:40.120 INFO:tasks.ceph.osd.6:Started 2023-09-30T01:57:40.120 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T01:57:40.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T01:57:40.122 INFO:tasks.ceph.osd.7:Started 2023-09-30T01:57:40.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:57:40.153 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.145+0000 7fdee75a4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:57:40.244 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 Processor -- start 2023-09-30T01:57:40.244 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 -- start start 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f41900a78d0 0x7f419013df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41900a7440 con 0x7f41900a9f30 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41900aa390 con 0x7f4190070760 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195d43700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41900a9270 con 0x7f41900a78d0 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418ffff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f41900a9f30 0x7f419013d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45244/0 (socket says 172.21.15.151:45244) 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418ffff700 1 -- 172.21.15.151:0/967081017 learned_addr learned my addr 172.21.15.151:0/967081017 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195542700 1 --2- 172.21.15.151:0/967081017 >> v2:172.21.15.172:3300/0 conn(0x7f41900a78d0 0x7f419013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224120351 0 0) 0x7f41900a7440 con 0x7f41900a9f30 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4178003570 con 0x7f41900a9f30 2023-09-30T01:57:40.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261186711 0 0) 0x7f41900aa390 con 0x7f4190070760 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41900a7440 con 0x7f4190070760 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2463553231 0 0) 0x7f4178003570 con 0x7f41900a9f30 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 >> v2:172.21.15.172:3300/0 conn(0x7f41900a78d0 msgr2=0x7f419013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 --2- 172.21.15.151:0/967081017 >> v2:172.21.15.172:3300/0 conn(0x7f41900a78d0 0x7f419013df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 >> v1:172.21.15.172:6789/0 conn(0x7f4190070760 legacy=0x7f419006b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4190070b30 con 0x7f41900a9f30 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f4180001470 con 0x7f41900a9f30 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 >> v1:172.21.15.151:6789/0 conn(0x7f41900a9f30 legacy=0x7f419013d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f4178004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178004810 0x7f4178004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4178005210 0x7f417800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41900a9270 con 0x7f4178004810 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417800c1b0 con 0x7f4178005210 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417800c370 con 0x7f4178003c40 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4194d41700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f4178004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f418ffff700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178004810 0x7f4178004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4195542700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4178005210 0x7f417800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.241+0000 7f4194d41700 1 -- 172.21.15.151:0/967081017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4178005210 msgr2=0x7f417800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4178005210 0x7f417800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 -- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178004810 msgr2=0x7f4178004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178004810 0x7f4178004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 -- 172.21.15.151:0/967081017 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178003570 con 0x7f4178003c40 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f418ffff700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4178004810 0x7f4178004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195542700 1 -- 172.21.15.151:0/967081017 reap_dead start 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f4178004030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41840019e0 tx=0x7f4184007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e39afe6e1b0c07d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400c5d0 con 0x7f4178003c40 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418400c730 con 0x7f4178003c40 2023-09-30T01:57:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f418f7fe700 1 -- 172.21.15.151:0/967081017 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400caf0 con 0x7f4178003c40 2023-09-30T01:57:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 msgr2=0x7f4178004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f4178004030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41840019e0 tx=0x7f4184007d30 comp rx=0 tx=0).stop 2023-09-30T01:57:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- 172.21.15.151:0/967081017 shutdown_connections 2023-09-30T01:57:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 --2- 172.21.15.151:0/967081017 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f4178004030 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- 172.21.15.151:0/967081017 >> 172.21.15.151:0/967081017 conn(0x7f41900b0030 msgr2=0x7f41900b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- 172.21.15.151:0/967081017 shutdown_connections 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- 172.21.15.151:0/967081017 wait complete. 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 Processor -- start 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- start start 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f419009e060 0x7f4190148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4190148bd0 0x7f419014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419009e650 con 0x7f419009e060 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419009f5a0 con 0x7f4190148bd0 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195d43700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419009f810 con 0x7f4178003c40 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:51284/0 (socket says 172.21.15.151:51284) 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f418ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f419009e060 0x7f4190148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4194d41700 1 -- 172.21.15.151:0/3470583156 learned_addr learned my addr 172.21.15.151:0/3470583156 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.245+0000 7f4195542700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4190148bd0 0x7f419014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 -- 172.21.15.151:0/3470583156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4190148bd0 msgr2=0x7f419014b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4190148bd0 0x7f419014b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 -- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f419009e060 msgr2=0x7f4190148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f419009e060 0x7f4190148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 -- 172.21.15.151:0/3470583156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419009c570 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4194d41700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4184012770 tx=0x7f4184015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400c5d0 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418400cea0 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4190149040 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184007b10 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419014c020 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4184019070 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f418401eae0 con 0x7f4178003c40 2023-09-30T01:57:40.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.249+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41900670a0 con 0x7f4178003c40 2023-09-30T01:57:40.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.253+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f418402c060 con 0x7f4178003c40 2023-09-30T01:57:40.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.317+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41900a94e0 con 0x7f4178003c40 2023-09-30T01:57:40.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.317+0000 7f418cff9700 1 -- 172.21.15.151:0/3470583156 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f4184022070 con 0x7f4178003c40 2023-09-30T01:57:40.323 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:40.323 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":10,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:40.080437+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 msgr2=0x7f419009d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4184012770 tx=0x7f4184015000 comp rx=0 tx=0).stop 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 shutdown_connections 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4190148bd0 0x7f419014b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f419009e060 0x7f4190148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 --2- 172.21.15.151:0/3470583156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4178003c40 0x7f419009d3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:40.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 >> 172.21.15.151:0/3470583156 conn(0x7f41900b0030 msgr2=0x7f419006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:40.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 shutdown_connections 2023-09-30T01:57:40.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:40.321+0000 7f4195d43700 1 -- 172.21.15.151:0/3470583156 wait complete. 2023-09-30T01:57:40.335 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T01:57:40.335 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T01:57:40.350 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.348+0000 7f5d93115000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:57:40.407 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.404+0000 7f5d93115000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:57:40.469 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.465+0000 7fdee75a4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:57:40.472 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.468+0000 7f5d93115000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:57:40.473 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-09-30T01:57:40.469+0000 7f05afaf33c0 -1 Falling back to public interface 2023-09-30T01:57:40.482 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-09-30T01:57:40.480+0000 7fc275a333c0 -1 Falling back to public interface 2023-09-30T01:57:40.490 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-09-30T01:57:40.488+0000 7efc2f4893c0 -1 Falling back to public interface 2023-09-30T01:57:40.490 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-09-30T01:57:40.488+0000 7efddf2a83c0 -1 Falling back to public interface 2023-09-30T01:57:40.496 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-09-30T01:57:40.493+0000 7fb6d2f823c0 -1 Falling back to public interface 2023-09-30T01:57:40.497 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-09-30T01:57:40.493+0000 7f2ada0943c0 -1 Falling back to public interface 2023-09-30T01:57:40.497 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-09-30T01:57:40.493+0000 7fbee1e393c0 -1 Falling back to public interface 2023-09-30T01:57:40.500 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-09-30T01:57:40.497+0000 7fbee0e02700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2023-09-30T01:57:40.630 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.628+0000 7f5d93115000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:57:40.633 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.629+0000 7fdee75a4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:57:40.695 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:40.692+0000 7f5d93115000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:57:40.698 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.693+0000 7fdee75a4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:57:40.734 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-09-30T01:57:40.732+0000 7f75335143c0 -1 Falling back to public interface 2023-09-30T01:57:40.750 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.745+0000 7fdee75a4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:57:40.806 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:40.801+0000 7fdee75a4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:57:41.086 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:41.081+0000 7fdee75a4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:57:41.141 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:57:41.168 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:41.141+0000 7fdee75a4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:57:41.210 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:41.205+0000 7fdee75a4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:57:41.255 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 Processor -- start 2023-09-30T01:57:41.256 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- start start 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4c64143ce0 0x7f4c641460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640ac0c0 con 0x7f4c640abad0 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6406df90 con 0x7f4c6406dbc0 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640a44e0 con 0x7f4c64143ce0 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6259c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4c640abad0 0x7f4c6413ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44616/0 (socket says 172.21.15.151:44616) 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6259c700 1 -- 172.21.15.151:0/3334056073 learned_addr learned my addr 172.21.15.151:0/3334056073 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:41.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6359e700 1 --2- 172.21.15.151:0/3334056073 >> v2:172.21.15.151:3300/0 conn(0x7f4c64143ce0 0x7f4c641460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:41.258 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309461290 0 0) 0x7f4c640ac0c0 con 0x7f4c640abad0 2023-09-30T01:57:41.258 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50003570 con 0x7f4c640abad0 2023-09-30T01:57:41.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6359e700 1 -- 172.21.15.151:0/3334056073 >> v1:172.21.15.172:6789/0 conn(0x7f4c6406dbc0 legacy=0x7f4c6413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715698281 0 0) 0x7f4c6406df90 con 0x7f4c6406dbc0 2023-09-30T01:57:41.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6359e700 1 -- 172.21.15.151:0/3334056073 >> v1:172.21.15.151:6789/0 conn(0x7f4c640abad0 legacy=0x7f4c6413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6359e700 1 -- 172.21.15.151:0/3334056073 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640ad240 con 0x7f4c64143ce0 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c6359e700 1 --2- 172.21.15.151:0/3334056073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c64143ce0 0x7f4c641460c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009910 tx=0x7f4c5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5189ef9d277df49e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c001470 con 0x7f4c64143ce0 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c011420 con 0x7f4c64143ce0 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c61d9b700 1 -- 172.21.15.151:0/3334056073 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0117c0 con 0x7f4c64143ce0 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- 172.21.15.151:0/3334056073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c64143ce0 msgr2=0x7f4c641460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 --2- 172.21.15.151:0/3334056073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c64143ce0 0x7f4c641460c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009910 tx=0x7f4c5c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- 172.21.15.151:0/3334056073 shutdown_connections 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 --2- 172.21.15.151:0/3334056073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c64143ce0 0x7f4c641460c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- 172.21.15.151:0/3334056073 >> 172.21.15.151:0/3334056073 conn(0x7f4c640b0c20 msgr2=0x7f4c640b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- 172.21.15.151:0/3334056073 shutdown_connections 2023-09-30T01:57:41.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.253+0000 7f4c691ba700 1 -- 172.21.15.151:0/3334056073 wait complete. 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 Processor -- start 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- start start 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c640abad0 0x7f4c6412efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 0x7f4c64131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:41.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64131000 con 0x7f4c6406dbc0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6414acb0 con 0x7f4c64143ce0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6414b710 con 0x7f4c640abad0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35386/0 (socket says 172.21.15.151:35386) 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 -- 172.21.15.151:0/4270606803 learned_addr learned my addr 172.21.15.151:0/4270606803 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c6259c700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c640abad0 0x7f4c6412efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c6359e700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 0x7f4c64131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 -- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c640abad0 msgr2=0x7f4c6412efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c640abad0 0x7f4c6412efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.264 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 -- 172.21.15.151:0/4270606803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 msgr2=0x7f4c64131360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 0x7f4c64131360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 -- 172.21.15.151:0/4270606803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c009420 con 0x7f4c6406dbc0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c6359e700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 0x7f4c64131360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c62d9d700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400cb60 tx=0x7f4c540017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5400a7e0 con 0x7f4c6406dbc0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5400a960 con 0x7f4c6406dbc0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6412f4e0 con 0x7f4c6406dbc0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5400ad20 con 0x7f4c6406dbc0 2023-09-30T01:57:41.265 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c64131920 con 0x7f4c6406dbc0 2023-09-30T01:57:41.266 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4c54003030 con 0x7f4c6406dbc0 2023-09-30T01:57:41.266 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.257+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f4c54012b80 con 0x7f4c6406dbc0 2023-09-30T01:57:41.266 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.261+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c640645d0 con 0x7f4c6406dbc0 2023-09-30T01:57:41.268 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.265+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4c54012de0 con 0x7f4c6406dbc0 2023-09-30T01:57:41.331 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.325+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c64138ce0 con 0x7f4c6406dbc0 2023-09-30T01:57:41.332 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c5affd700 1 -- 172.21.15.151:0/4270606803 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f4c54012630 con 0x7f4c6406dbc0 2023-09-30T01:57:41.332 INFO:teuthology.misc.health.smithi151.stdout: 2023-09-30T01:57:41.333 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:40.080437+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:57:41.334 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 msgr2=0x7f4c6414b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:41.334 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400cb60 tx=0x7f4c540017a0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.334 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 shutdown_connections 2023-09-30T01:57:41.334 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c64143ce0 0x7f4c64131360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.334 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c640abad0 0x7f4c6412efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.335 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 --2- 172.21.15.151:0/4270606803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c6406dbc0 0x7f4c6414b1d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:41.335 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 >> 172.21.15.151:0/4270606803 conn(0x7f4c640b0c20 msgr2=0x7f4c640ad4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:41.335 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 shutdown_connections 2023-09-30T01:57:41.335 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:41.329+0000 7f4c691ba700 1 -- 172.21.15.151:0/4270606803 wait complete. 2023-09-30T01:57:41.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T01:57:41.376 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:41.373+0000 7fdee75a4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:57:41.396 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:41.392+0000 7f5d93115000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:57:41.445 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:41.441+0000 7fdee75a4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:57:41.458 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:41.456+0000 7f5d709f6700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:57:41.458 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:41.456+0000 7f5d709f6700 -1 pg_autoscaler.serve: 2023-09-30T01:57:41.459 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:41.456+0000 7f5d709f6700 -1 Traceback (most recent call last): 2023-09-30T01:57:41.459 INFO:tasks.ceph.mgr.x.smithi172.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T01:57:41.459 INFO:tasks.ceph.mgr.x.smithi172.stderr: self._maybe_adjust() 2023-09-30T01:57:41.459 INFO:tasks.ceph.mgr.x.smithi172.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T01:57:41.460 INFO:tasks.ceph.mgr.x.smithi172.stderr: if self.noautoscale: 2023-09-30T01:57:41.460 INFO:tasks.ceph.mgr.x.smithi172.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:57:41.460 INFO:tasks.ceph.mgr.x.smithi172.stderr: 2023-09-30T01:57:42.179 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-09-30T01:57:42.173+0000 7fdee75a4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:57:42.447 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:42.444+0000 7f5d7c20d700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T01:57:42.448 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-09-30T01:57:42.444+0000 7f5d7c20d700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T01:57:44.068 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-09-30T01:57:44.064+0000 7efddf2a83c0 -1 osd.5 0 log_to_monitors true 2023-09-30T01:57:44.124 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-09-30T01:57:44.120+0000 7fc275a333c0 -1 osd.4 0 log_to_monitors true 2023-09-30T01:57:44.151 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-09-30T01:57:44.145+0000 7f2ada0943c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:57:44.288 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-09-30T01:57:44.284+0000 7efc2f4893c0 -1 osd.7 0 log_to_monitors true 2023-09-30T01:57:44.291 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-09-30T01:57:44.285+0000 7f05afaf33c0 -1 osd.3 0 log_to_monitors true 2023-09-30T01:57:44.393 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-09-30T01:57:44.389+0000 7fb6d2f823c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:57:44.407 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-09-30T01:57:44.405+0000 7fbee1e393c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:57:44.495 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-09-30T01:57:44.492+0000 7f75335143c0 -1 osd.6 0 log_to_monitors true 2023-09-30T01:57:45.462 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-09-30T01:57:45.460+0000 7fc2711d3700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T01:57:45.463 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-09-30T01:57:45.460+0000 7efddaa48700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T01:57:45.464 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-09-30T01:57:45.461+0000 7f2ad7059700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T01:57:45.464 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-09-30T01:57:45.460+0000 7efc2c44e700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T01:57:45.464 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-09-30T01:57:45.461+0000 7fbedd5d9700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T01:57:45.466 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-09-30T01:57:45.461+0000 7fb6ce722700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T01:57:45.467 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-09-30T01:57:45.465+0000 7f05acab8700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T01:57:45.473 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-09-30T01:57:45.469+0000 7fbed8381700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.474 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-09-30T01:57:45.469+0000 7f2acfddb700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.474 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-09-30T01:57:45.469+0000 7fb6c94ca700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.474 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-09-30T01:57:45.469+0000 7f05a683c700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.475 INFO:tasks.ceph.osd.5.smithi172.stderr:2023-09-30T01:57:45.472+0000 7efdd57f0700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.475 INFO:tasks.ceph.osd.7.smithi172.stderr:2023-09-30T01:57:45.472+0000 7efc261d2700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:45.475 INFO:tasks.ceph.osd.4.smithi172.stderr:2023-09-30T01:57:45.472+0000 7fc26bf7b700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:46.471 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-09-30T01:57:46.468+0000 7f752ecb4700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T01:57:46.478 INFO:tasks.ceph.osd.6.smithi172.stderr:2023-09-30T01:57:46.476+0000 7f752a25d700 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:57:46.498 INFO:tasks.ceph.mon.a.smithi151.stderr:2023-09-30T01:57:46.493+0000 7f0d2a3f0700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:57:48.150 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:57:48.251 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.244+0000 7f5c96956700 1 Processor -- start 2023-09-30T01:57:48.251 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.244+0000 7f5c96956700 1 -- start start 2023-09-30T01:57:48.252 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5c900ab3d0 0x7f5c9013a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.252 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5c9006b350 0x7f5c9006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.252 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9006d190 con 0x7f5c900ab7a0 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9006d890 con 0x7f5c9006b350 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9006e320 con 0x7f5c900ab3d0 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5c9006b350 0x7f5c9006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5c9006b350 0x7f5c9006dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35444/0 (socket says 172.21.15.151:35444) 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/686421041 learned_addr learned my addr 172.21.15.151:0/686421041 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:48.253 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8ffff700 1 --2- 172.21.15.151:0/686421041 >> v2:172.21.15.172:3300/0 conn(0x7f5c900ab3d0 0x7f5c9013a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/686421041 >> v2:172.21.15.172:3300/0 conn(0x7f5c900ab3d0 msgr2=0x7f5c9013a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 --2- 172.21.15.151:0/686421041 >> v2:172.21.15.172:3300/0 conn(0x7f5c900ab3d0 0x7f5c9013a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/686421041 >> v1:172.21.15.172:6789/0 conn(0x7f5c900ab7a0 legacy=0x7f5c900a9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/686421041 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9006ae10 con 0x7f5c9006b350 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8f7fe700 1 --2- 172.21.15.151:0/686421041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9006dde0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c90069750 tx=0x7f5c80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4612c4d3dd4d15 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8effd700 1 -- 172.21.15.151:0/686421041 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000e070 con 0x7f5c9006b350 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8effd700 1 -- 172.21.15.151:0/686421041 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8000ba20 con 0x7f5c9006b350 2023-09-30T01:57:48.254 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c8effd700 1 -- 172.21.15.151:0/686421041 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c800092e0 con 0x7f5c9006b350 2023-09-30T01:57:48.255 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- 172.21.15.151:0/686421041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 msgr2=0x7f5c9006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.255 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 --2- 172.21.15.151:0/686421041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9006dde0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c90069750 tx=0x7f5c80009840 comp rx=0 tx=0).stop 2023-09-30T01:57:48.255 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- 172.21.15.151:0/686421041 shutdown_connections 2023-09-30T01:57:48.255 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 --2- 172.21.15.151:0/686421041 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9006dde0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.256 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 --2- 172.21.15.151:0/686421041 >> v2:172.21.15.172:3300/0 conn(0x7f5c900ab3d0 0x7f5c9013a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.256 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- 172.21.15.151:0/686421041 >> 172.21.15.151:0/686421041 conn(0x7f5c90067020 msgr2=0x7f5c90067420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:48.256 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- 172.21.15.151:0/686421041 shutdown_connections 2023-09-30T01:57:48.256 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.248+0000 7f5c96956700 1 -- 172.21.15.151:0/686421041 wait complete. 2023-09-30T01:57:48.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 Processor -- start 2023-09-30T01:57:48.257 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 -- start start 2023-09-30T01:57:48.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.259 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900ab7a0 0x7f5c9014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9014c000 con 0x7f5c9006b350 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9013e3d0 con 0x7f5c900ab7a0 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c96956700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900a5d70 con 0x7f5c900ab3d0 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53416/0 (socket says 172.21.15.151:53416) 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35452/0 (socket says 172.21.15.151:35452) 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/1148037395 learned_addr learned my addr 172.21.15.151:0/1148037395 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c94ef3700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900ab7a0 0x7f5c9014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900ab7a0 msgr2=0x7f5c9014c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.260 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900ab7a0 0x7f5c9014c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 msgr2=0x7f5c9013d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 -- 172.21.15.151:0/1148037395 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c80009ae0 con 0x7f5c900ab3d0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8ffff700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c8f7fe700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c90069780 tx=0x7f5c80018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c800078f0 con 0x7f5c900ab3d0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.252+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c80007a70 con 0x7f5c900ab3d0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c900a9300 con 0x7f5c900ab3d0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80007e30 con 0x7f5c900ab3d0 2023-09-30T01:57:48.261 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9014cbf0 con 0x7f5c900ab3d0 2023-09-30T01:57:48.262 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9014ce10 con 0x7f5c900ab3d0 2023-09-30T01:57:48.262 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7f5c80003040 con 0x7f5c900ab3d0 2023-09-30T01:57:48.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c87fff700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 0x7f5c70062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4842+0+0 (secure 0 0 0) 0x7f5c8007de00 con 0x7f5c900ab3d0 2023-09-30T01:57:48.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.256+0000 7f5c8ffff700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 0x7f5c70062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.263 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.260+0000 7f5c8ffff700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 0x7f5c70062970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800bfd0 tx=0x7f5c7800b430 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.268 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.264+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c8004e830 con 0x7f5c900ab3d0 2023-09-30T01:57:48.428 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.424+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c900a9680 con 0x7f5c900ab3d0 2023-09-30T01:57:48.429 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.424+0000 7f5c87fff700 1 -- 172.21.15.151:0/1148037395 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+12156 (secure 0 0 0) 0x7f5c80051e50 con 0x7f5c900ab3d0 2023-09-30T01:57:48.430 INFO:teuthology.misc.health.smithi151.stdout: 2023-09-30T01:57:48.430 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":14,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:47.500496+0000","last_up_change":"2023-09-30T01:57:47.500496+0000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:57:47.499184+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":31361},{"type":"v1","addr":"172.21.15.151:6818","nonce":31361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":31361},{"type":"v1","addr":"172.21.15.151:6824","nonce":31361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":31361},{"type":"v1","addr":"172.21.15.151:6831","nonce":31361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":31361},{"type":"v1","addr":"172.21.15.151:6828","nonce":31361}]},"public_addr":"172.21.15.151:6818/31361","cluster_addr":"172.21.15.151:6824/31361","heartbeat_back_addr":"172.21.15.151:6831/31361","heartbeat_front_addr":"172.21.15.151:6828/31361","state":["exists","up"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":31363},{"type":"v1","addr":"172.21.15.151:6811","nonce":31363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":31363},{"type":"v1","addr":"172.21.15.151:6815","nonce":31363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":31363},{"type":"v1","addr":"172.21.15.151:6823","nonce":31363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":31363},{"type":"v1","addr":"172.21.15.151:6819","nonce":31363}]},"public_addr":"172.21.15.151:6811/31363","cluster_addr":"172.21.15.151:6815/31363","heartbeat_back_addr":"172.21.15.151:6823/31363","heartbeat_front_addr":"172.21.15.151:6819/31363","state":["exists","up"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":31362},{"type":"v1","addr":"172.21.15.151:6810","nonce":31362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":31362},{"type":"v1","addr":"172.21.15.151:6816","nonce":31362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":31362},{"type":"v1","addr":"172.21.15.151:6829","nonce":31362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":31362},{"type":"v1","addr":"172.21.15.151:6825","nonce":31362}]},"public_addr":"172.21.15.151:6810/31362","cluster_addr":"172.21.15.151:6816/31362","heartbeat_back_addr":"172.21.15.151:6829/31362","heartbeat_front_addr":"172.21.15.151:6825/31362","state":["exists","up"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":31360},{"type":"v1","addr":"172.21.15.151:6801","nonce":31360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":31360},{"type":"v1","addr":"172.21.15.151:6803","nonce":31360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":31360},{"type":"v1","addr":"172.21.15.151:6807","nonce":31360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":31360},{"type":"v1","addr":"172.21.15.151:6805","nonce":31360}]},"public_addr":"172.21.15.151:6801/31360","cluster_addr":"172.21.15.151:6803/31360","heartbeat_back_addr":"172.21.15.151:6807/31360","heartbeat_front_addr":"172.21.15.151:6805/31360","state":["exists","up"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":31451},{"type":"v1","addr":"172.21.15.172:6801","nonce":31451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":31451},{"type":"v1","addr":"172.21.15.172:6803","nonce":31451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":31451},{"type":"v1","addr":"172.21.15.172:6807","nonce":31451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":31451},{"type":"v1","addr":"172.21.15.172:6805","nonce":31451}]},"public_addr":"172.21.15.172:6801/31451","cluster_addr":"172.21.15.172:6803/31451","heartbeat_back_addr":"172.21.15.172:6807/31451","heartbeat_front_addr":"172.21.15.172:6805/31451","state":["exists","up"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":31452},{"type":"v1","addr":"172.21.15.172:6811","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":31452},{"type":"v1","addr":"172.21.15.172:6814","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":31452},{"type":"v1","addr":"172.21.15.172:6820","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":31452},{"type":"v1","addr":"172.21.15.172:6817","nonce":31452}]},"public_addr":"172.21.15.172:6811/31452","cluster_addr":"172.21.15.172:6814/31452","heartbeat_back_addr":"172.21.15.172:6820/31452","heartbeat_front_addr":"172.21.15.172:6817/31452","state":["exists","up"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31453},{"type":"v1","addr":"172.21.15.172:6825","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":31453},{"type":"v1","addr":"172.21.15.172:6827","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":31453},{"type":"v1","addr":"172.21.15.172:6831","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":31453},{"type":"v1","addr":"172.21.15.172:6829","nonce":31453}]},"public_addr":"172.21.15.172:6825/31453","cluster_addr":"172.21.15.172:6827/31453","heartbeat_back_addr":"172.21.15.172:6831/31453","heartbeat_front_addr":"172.21.15.172:6829/31453","state":["exists","up"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":31450},{"type":"v1","addr":"172.21.15.172:6809","nonce":31450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":31450},{"type":"v1","addr":"172.21.15.172:6815","nonce":31450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":31450},{"type":"v1","addr":"172.21.15.172:6823","nonce":31450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":31450},{"type":"v1","addr":"172.21.15.172:6821","nonce":31450}]},"public_addr":"172.21.15.172:6809/31450","cluster_addr":"172.21.15.172:6815/31450","heartbeat_back_addr":"172.21.15.172:6823/31450","heartbeat_front_addr":"172.21.15.172:6821/31450","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.403867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.433944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.158817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.278191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.087404+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.063855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.286608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:57:48.432 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 msgr2=0x7f5c70062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.432 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 0x7f5c70062970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800bfd0 tx=0x7f5c7800b430 comp rx=0 tx=0).stop 2023-09-30T01:57:48.432 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 msgr2=0x7f5c900a8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c90069780 tx=0x7f5c80018000 comp rx=0 tx=0).stop 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 shutdown_connections 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5c700604c0 0x7f5c70062970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c900ab7a0 0x7f5c9014c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c900ab3d0 0x7f5c900a8dc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 --2- 172.21.15.151:0/1148037395 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c9006b350 0x7f5c9013d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 >> 172.21.15.151:0/1148037395 conn(0x7f5c90067020 msgr2=0x7f5c900aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 shutdown_connections 2023-09-30T01:57:48.433 INFO:teuthology.misc.health.smithi151.stderr:2023-09-30T01:57:48.428+0000 7f5c96956700 1 -- 172.21.15.151:0/1148037395 wait complete. 2023-09-30T01:57:48.447 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T01:57:48.447 INFO:tasks.ceph:Creating RBD pool 2023-09-30T01:57:48.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T01:57:48.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.544+0000 7f96ba1b5700 1 Processor -- start 2023-09-30T01:57:48.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- start start 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f96b40708e0 0x7f96b413db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b413e0c0 con 0x7f96b4070510 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b40ac2f0 con 0x7f96b40aa4c0 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b40ac5d0 con 0x7f96b40708e0 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f96b40708e0 0x7f96b413db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b91b3700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f96b4070510 0x7f96b4136ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44628/0 (socket says 172.21.15.151:44628) 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f96b40708e0 0x7f96b413db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35464/0 (socket says 172.21.15.151:35464) 2023-09-30T01:57:48.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b91b3700 1 -- 172.21.15.151:0/3054738980 learned_addr learned my addr 172.21.15.151:0/3054738980 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b3fff700 1 -- 172.21.15.151:0/3054738980 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951615170 0 0) 0x7f96b413e0c0 con 0x7f96b4070510 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b3fff700 1 -- 172.21.15.151:0/3054738980 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c003570 con 0x7f96b4070510 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 -- 172.21.15.151:0/3054738980 >> v1:172.21.15.172:6789/0 conn(0x7f96b40aa4c0 legacy=0x7f96b413d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 -- 172.21.15.151:0/3054738980 >> v1:172.21.15.151:6789/0 conn(0x7f96b4070510 legacy=0x7f96b4136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 -- 172.21.15.151:0/3054738980 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40ab2c0 con 0x7f96b40708e0 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b99b4700 1 --2- 172.21.15.151:0/3054738980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b40708e0 0x7f96b413db80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0098e0 tx=0x7f96ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77107e005b407005 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b3fff700 1 -- 172.21.15.151:0/3054738980 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00b640 con 0x7f96b40708e0 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b3fff700 1 -- 172.21.15.151:0/3054738980 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac00b7a0 con 0x7f96b40708e0 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96b3fff700 1 -- 172.21.15.151:0/3054738980 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00bb40 con 0x7f96b40708e0 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3054738980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b40708e0 msgr2=0x7f96b413db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3054738980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b40708e0 0x7f96b413db80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0098e0 tx=0x7f96ac0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3054738980 shutdown_connections 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3054738980 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b40708e0 0x7f96b413db80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3054738980 >> 172.21.15.151:0/3054738980 conn(0x7f96b40b0400 msgr2=0x7f96b40b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3054738980 shutdown_connections 2023-09-30T01:57:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.548+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3054738980 wait complete. 2023-09-30T01:57:48.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 Processor -- start 2023-09-30T01:57:48.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- start start 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96b40708e0 0x7f96b4150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 0x7f96b4150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4150c90 con 0x7f96b4070510 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4140b30 con 0x7f96b40708e0 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b413efa0 con 0x7f96b40aa4c0 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b99b4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 0x7f96b4150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35480/0 (socket says 172.21.15.151:35480) 2023-09-30T01:57:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 -- 172.21.15.151:0/3588434929 learned_addr learned my addr 172.21.15.151:0/3588434929 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b99b4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 0x7f96b4150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53424/0 (socket says 172.21.15.151:53424) 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b89b2700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96b40708e0 0x7f96b4150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 -- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 msgr2=0x7f96b4150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 0x7f96b4150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 -- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96b40708e0 msgr2=0x7f96b4150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96b40708e0 0x7f96b4150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 -- 172.21.15.151:0/3588434929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009420 con 0x7f96b4070510 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b91b3700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96a400cd30 tx=0x7f96a4001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400ce90 con 0x7f96b4070510 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a400aac0 con 0x7f96b4070510 2023-09-30T01:57:48.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b413f280 con 0x7f96b4070510 2023-09-30T01:57:48.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400ae60 con 0x7f96b4070510 2023-09-30T01:57:48.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.552+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b406da90 con 0x7f96b4070510 2023-09-30T01:57:48.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7f96a4007300 con 0x7f96b4070510 2023-09-30T01:57:48.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b40008d0 con 0x7f96b4070510 2023-09-30T01:57:48.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96b17fa700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 0x7f96a0062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:48.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f96a407c6e0 con 0x7f96b4070510 2023-09-30T01:57:48.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96b89b2700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 0x7f96a0062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:48.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.556+0000 7f96b89b2700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 0x7f96a0062970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96b406d700 tx=0x7f96a8007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:48.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.560+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f96a404d180 con 0x7f96b4070510 2023-09-30T01:57:48.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:48.716+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f96b4140ca0 con 0x7f96b4070510 2023-09-30T01:57:49.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.532+0000 7f96b17fa700 1 -- 172.21.15.151:0/3588434929 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7f96a40507a0 con 0x7f96b4070510 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 msgr2=0x7f96a0062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 0x7f96a0062970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96b406d700 tx=0x7f96a8007ec0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 msgr2=0x7f96b41411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96a400cd30 tx=0x7f96a4001b10 comp rx=0 tx=0).stop 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 shutdown_connections 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f96a00604c0 0x7f96a0062970 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96b40aa4c0 0x7f96b4150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f96b40708e0 0x7f96b4150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 --2- 172.21.15.151:0/3588434929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96b4070510 0x7f96b41411f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 >> 172.21.15.151:0/3588434929 conn(0x7f96b40b0400 msgr2=0x7f96b4136a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 shutdown_connections 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.536+0000 7f96ba1b5700 1 -- 172.21.15.151:0/3588434929 wait complete. 2023-09-30T01:57:49.542 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'rbd' created 2023-09-30T01:57:49.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T01:57:49.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 Processor -- start 2023-09-30T01:57:49.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 -- start start 2023-09-30T01:57:49.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7feea80a9a20 0x7feea813ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80aa070 con 0x7feea80675a0 2023-09-30T01:57:49.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea8067970 con 0x7feea80aa500 2023-09-30T01:57:49.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea7fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea80a9320 con 0x7feea80a9a20 2023-09-30T01:57:49.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea6ffd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7feea80675a0 0x7feea81409d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44644/0 (socket says 172.21.15.151:44644) 2023-09-30T01:57:49.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea6ffd700 1 -- 172.21.15.151:0/1510709175 learned_addr learned my addr 172.21.15.151:0/1510709175 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:49.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea77fe700 1 --2- 172.21.15.151:0/1510709175 >> v2:172.21.15.172:3300/0 conn(0x7feea80a9a20 0x7feea813ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267731782 0 0) 0x7feea80aa070 con 0x7feea80675a0 2023-09-30T01:57:49.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.652+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90003570 con 0x7feea80675a0 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea77fe700 1 -- 172.21.15.151:0/1510709175 >> v1:172.21.15.172:6789/0 conn(0x7feea80aa500 legacy=0x7feea813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555011414 0 0) 0x7fee90003570 con 0x7feea80675a0 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea77fe700 1 -- 172.21.15.151:0/1510709175 >> v1:172.21.15.151:6789/0 conn(0x7feea80675a0 legacy=0x7feea81409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea77fe700 1 -- 172.21.15.151:0/1510709175 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80a9600 con 0x7feea80a9a20 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea77fe700 1 --2- 172.21.15.151:0/1510709175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80a9a20 0x7feea813ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feea00098e0 tx=0x7feea00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d476bca42cf90c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980017f0 con 0x7feea80a9a20 2023-09-30T01:57:49.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea0011420 con 0x7feea80a9a20 2023-09-30T01:57:49.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea5ffb700 1 -- 172.21.15.151:0/1510709175 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00117a0 con 0x7feea80a9a20 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 -- 172.21.15.151:0/1510709175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80a9a20 msgr2=0x7feea813ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 --2- 172.21.15.151:0/1510709175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80a9a20 0x7feea813ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feea00098e0 tx=0x7feea00091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 -- 172.21.15.151:0/1510709175 shutdown_connections 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 --2- 172.21.15.151:0/1510709175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80a9a20 0x7feea813ded0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 -- 172.21.15.151:0/1510709175 >> 172.21.15.151:0/1510709175 conn(0x7feea80b0920 msgr2=0x7feea80b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 -- 172.21.15.151:0/1510709175 shutdown_connections 2023-09-30T01:57:49.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 -- 172.21.15.151:0/1510709175 wait complete. 2023-09-30T01:57:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.656+0000 7feea7fff700 1 Processor -- start 2023-09-30T01:57:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 -- start start 2023-09-30T01:57:49.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80675a0 0x7feea811cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea811a6c0 con 0x7feea80a9a20 2023-09-30T01:57:49.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea811f080 con 0x7feea80675a0 2023-09-30T01:57:49.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea7fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea811f2f0 con 0x7feea80aa500 2023-09-30T01:57:49.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.660+0000 7feea67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea67fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35482/0 (socket says 172.21.15.151:35482) 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53436/0 (socket says 172.21.15.151:53436) 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea67fc700 1 -- 172.21.15.151:0/779138464 learned_addr learned my addr 172.21.15.151:0/779138464 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea6ffd700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80675a0 0x7feea811cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 -- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80675a0 msgr2=0x7feea811cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80675a0 0x7feea811cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 -- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 msgr2=0x7feea814cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 -- 172.21.15.151:0/779138464 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0009420 con 0x7feea80aa500 2023-09-30T01:57:49.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea67fc700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:49.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea77fe700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feea8069930 tx=0x7feea0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea001a660 con 0x7feea80aa500 2023-09-30T01:57:49.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea001a7e0 con 0x7feea80aa500 2023-09-30T01:57:49.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea001aba0 con 0x7feea80aa500 2023-09-30T01:57:49.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea814d690 con 0x7feea80aa500 2023-09-30T01:57:49.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea811dc90 con 0x7feea80aa500 2023-09-30T01:57:49.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea814c930 con 0x7feea80aa500 2023-09-30T01:57:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.664+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7feea002d440 con 0x7feea80aa500 2023-09-30T01:57:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.668+0000 7fee8effd700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 0x7fee94062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.668+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5195+0+0 (secure 0 0 0) 0x7feea007e300 con 0x7feea80aa500 2023-09-30T01:57:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.668+0000 7feea6ffd700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 0x7fee94062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.668+0000 7feea6ffd700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 0x7fee94062970 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fee98007fd0 tx=0x7fee98000f40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.672+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feea814c930 con 0x7feea80aa500 2023-09-30T01:57:49.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:49.840+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feea814c930 con 0x7feea80aa500 2023-09-30T01:57:50.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.532+0000 7fee8effd700 1 -- 172.21.15.151:0/779138464 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7feea814c930 con 0x7feea80aa500 2023-09-30T01:57:50.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 msgr2=0x7fee94062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 0x7fee94062970 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fee98007fd0 tx=0x7fee98000f40 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 msgr2=0x7feea814d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feea8069930 tx=0x7feea0016000 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 shutdown_connections 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fee940604c0 0x7fee94062970 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feea80aa500 0x7feea814d150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feea80a9a20 0x7feea814cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 --2- 172.21.15.151:0/779138464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feea80675a0 0x7feea811cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.536+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 >> 172.21.15.151:0/779138464 conn(0x7feea80b0920 msgr2=0x7feea80b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:50.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.540+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 shutdown_connections 2023-09-30T01:57:50.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.540+0000 7feea7fff700 1 -- 172.21.15.151:0/779138464 wait complete. 2023-09-30T01:57:50.546 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T01:57:50.557 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T01:57:50.557 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T01:57:50.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T01:57:50.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T01:57:50.558 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T01:57:50.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 Processor -- start 2023-09-30T01:57:50.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- start start 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 0x7f7d30144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3006c310 con 0x7f7d300b0a80 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3006c030 con 0x7f7d301427e0 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300b0e50 con 0x7f7d300aa4a0 2023-09-30T01:57:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2ed9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 0x7f7d30144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44392/0 (socket says 172.21.15.151:44392) 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/544060008 learned_addr learned my addr 172.21.15.151:0/544060008 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/544060008 >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 msgr2=0x7f7d30144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/544060008 >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 0x7f7d30144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/544060008 >> v1:172.21.15.172:6789/0 conn(0x7f7d300b0a80 legacy=0x7f7d3006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/544060008 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300ace90 con 0x7f7d300aa4a0 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2ed9d700 1 --2- 172.21.15.151:0/544060008 >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 0x7f7d30144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/544060008 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009f70 tx=0x7f7d28009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e71e03e52bc3c2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2d59a700 1 -- 172.21.15.151:0/544060008 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800e070 con 0x7f7d300aa4a0 2023-09-30T01:57:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2d59a700 1 -- 172.21.15.151:0/544060008 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2800bab0 con 0x7f7d300aa4a0 2023-09-30T01:57:50.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2d59a700 1 -- 172.21.15.151:0/544060008 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280092e0 con 0x7f7d300aa4a0 2023-09-30T01:57:50.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- 172.21.15.151:0/544060008 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 msgr2=0x7f7d300aa8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/544060008 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009f70 tx=0x7f7d28009840 comp rx=0 tx=0).stop 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- 172.21.15.151:0/544060008 shutdown_connections 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/544060008 >> v2:172.21.15.151:3300/0 conn(0x7f7d301427e0 0x7f7d30144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/544060008 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d300aa4a0 0x7f7d300aa8d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- 172.21.15.151:0/544060008 >> 172.21.15.151:0/544060008 conn(0x7f7d30070510 msgr2=0x7f7d30070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- 172.21.15.151:0/544060008 shutdown_connections 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.656+0000 7f7d2f59e700 1 -- 172.21.15.151:0/544060008 wait complete. 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 Processor -- start 2023-09-30T01:57:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- start start 2023-09-30T01:57:50.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d300aa4a0 0x7f7d301300e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 0x7f7d3013cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 0x7f7d3013a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30137ed0 con 0x7f7d300b0a80 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30139d90 con 0x7f7d300aa4a0 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3013c0a0 con 0x7f7d301427e0 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 0x7f7d3013a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 0x7f7d3013a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44406/0 (socket says 172.21.15.151:44406) 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2ed9d700 1 -- 172.21.15.151:0/272862959 learned_addr learned my addr 172.21.15.151:0/272862959 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 0x7f7d3013cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2e59c700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d300aa4a0 0x7f7d301300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 msgr2=0x7f7d3013a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 0x7f7d3013a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d300aa4a0 msgr2=0x7f7d301300e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d300aa4a0 0x7f7d301300e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 -- 172.21.15.151:0/272862959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20009420 con 0x7f7d300b0a80 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2dd9b700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 0x7f7d3013cce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d28015000 tx=0x7f7d28018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28008a00 con 0x7f7d300b0a80 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28008b60 con 0x7f7d300b0a80 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d28009ae0 con 0x7f7d300b0a80 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2801d420 con 0x7f7d300b0a80 2023-09-30T01:57:50.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.660+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d30138580 con 0x7f7d300b0a80 2023-09-30T01:57:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d30138080 con 0x7f7d300b0a80 2023-09-30T01:57:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7f7d28003040 con 0x7f7d300b0a80 2023-09-30T01:57:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d267fc700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 0x7f7d10062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7d2807e1d0 con 0x7f7d300b0a80 2023-09-30T01:57:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d2e59c700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 0x7f7d10062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:50.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.664+0000 7f7d2e59c700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 0x7f7d10062970 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d18003a80 tx=0x7f7d18007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.672+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7d30138080 con 0x7f7d300b0a80 2023-09-30T01:57:50.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.872+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7d30138080 con 0x7f7d300b0a80 2023-09-30T01:57:50.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d267fc700 1 -- 172.21.15.151:0/272862959 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+154044 (secure 0 0 0) 0x7f7d30138080 con 0x7f7d300b0a80 2023-09-30T01:57:50.886 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:50.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 msgr2=0x7f7d10062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 0x7f7d10062970 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d18003a80 tx=0x7f7d18007ec0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 msgr2=0x7f7d3013cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 0x7f7d3013cce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d28015000 tx=0x7f7d28018000 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 shutdown_connections 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7d100604c0 0x7f7d10062970 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d301427e0 0x7f7d3013a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d300b0a80 0x7f7d3013cce0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 --2- 172.21.15.151:0/272862959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d300aa4a0 0x7f7d301300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 >> 172.21.15.151:0/272862959 conn(0x7f7d30070510 msgr2=0x7f7d3006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 shutdown_connections 2023-09-30T01:57:50.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:50.880+0000 7f7d2f59e700 1 -- 172.21.15.151:0/272862959 wait complete. 2023-09-30T01:57:50.899 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":31144},{"type":"v1","addr":"172.21.15.172:6833","nonce":31144}]},"active_addr":"172.21.15.172:6833/31144","active_change":"2023-09-30T01:57:41.415604+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2038181573}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2749365295}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1347550926}]}]} 2023-09-30T01:57:50.904 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T01:57:50.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T01:57:50.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:57:51.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 Processor -- start 2023-09-30T01:57:51.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- start start 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02940b0a80 0x7f029406e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f02940aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 0x7f02940aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029406c3f0 con 0x7f02940b0a80 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029406db10 con 0x7f0294139f20 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02940b0e50 con 0x7f029406ea20 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02940b0a80 0x7f029406e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02940b0a80 0x7f029406e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55400/0 (socket says 172.21.15.151:55400) 2023-09-30T01:57:51.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 -- 172.21.15.151:0/643871719 learned_addr learned my addr 172.21.15.151:0/643871719 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f0292d9d700 1 --2- 172.21.15.151:0/643871719 >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 0x7f02940aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f0291d9b700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f02940aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 -- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 msgr2=0x7f02940aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f02940aa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 -- 172.21.15.151:0/643871719 >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 msgr2=0x7f02940aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 --2- 172.21.15.151:0/643871719 >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 0x7f02940aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 -- 172.21.15.151:0/643871719 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029406dd80 con 0x7f02940b0a80 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f0292d9d700 1 --2- 172.21.15.151:0/643871719 >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 0x7f02940aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029259c700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f029406e4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f029413a3a0 tx=0x7f028400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3296cd28ca94000b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029159a700 1 -- 172.21.15.151:0/643871719 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028400dba0 con 0x7f02940b0a80 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029159a700 1 -- 172.21.15.151:0/643871719 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028400dd00 con 0x7f02940b0a80 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029159a700 1 -- 172.21.15.151:0/643871719 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840196d0 con 0x7f02940b0a80 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 msgr2=0x7f029406e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f029406e4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f029413a3a0 tx=0x7f028400b4f0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- 172.21.15.151:0/643871719 shutdown_connections 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- 172.21.15.151:0/643871719 >> v2:172.21.15.172:3300/0 conn(0x7f0294139f20 0x7f02940aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f02940aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 --2- 172.21.15.151:0/643871719 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f029406e4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.004+0000 7f029359e700 1 -- 172.21.15.151:0/643871719 >> 172.21.15.151:0/643871719 conn(0x7f0294070510 msgr2=0x7f0294070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- 172.21.15.151:0/643871719 shutdown_connections 2023-09-30T01:57:51.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- 172.21.15.151:0/643871719 wait complete. 2023-09-30T01:57:51.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 Processor -- start 2023-09-30T01:57:51.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- start start 2023-09-30T01:57:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f029413d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 0x7f029414cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294118530 con 0x7f02940b0a80 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02941193a0 con 0x7f0294139f20 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02941373b0 con 0x7f029406ea20 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55408/0 (socket says 172.21.15.151:55408) 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 -- 172.21.15.151:0/2477407927 learned_addr learned my addr 172.21.15.151:0/2477407927 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029259c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f029413d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0292d9d700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 0x7f029414cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 -- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 msgr2=0x7f029413d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f029413d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 -- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 msgr2=0x7f029414cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 0x7f029414cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 -- 172.21.15.151:0/2477407927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c009a20 con 0x7f02940b0a80 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0292d9d700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 0x7f029414cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f0291d9b700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f028800d890 tx=0x7f0288008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02880125f0 con 0x7f02940b0a80 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0288012770 con 0x7f02940b0a80 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028400b790 con 0x7f02940b0a80 2023-09-30T01:57:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.008+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288012b30 con 0x7f02940b0a80 2023-09-30T01:57:51.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0294119080 con 0x7f02940b0a80 2023-09-30T01:57:51.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7f028801e460 con 0x7f02940b0a80 2023-09-30T01:57:51.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02940008d0 con 0x7f02940b0a80 2023-09-30T01:57:51.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f02827fc700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 0x7f0274062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f028807cbd0 con 0x7f02940b0a80 2023-09-30T01:57:51.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.012+0000 7f029259c700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 0x7f0274062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.016+0000 7f029259c700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 0x7f0274062970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f029413a3a0 tx=0x7f028401b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.020+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f028804d3f0 con 0x7f02940b0a80 2023-09-30T01:57:51.187 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.180+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f029406cbf0 con 0x7f02940b0a80 2023-09-30T01:57:51.188 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f02827fc700 1 -- 172.21.15.151:0/2477407927 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13708 (secure 0 0 0) 0x7f028807ce60 con 0x7f02940b0a80 2023-09-30T01:57:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:51.189 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":17,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:50.527260+0000","last_up_change":"2023-09-30T01:57:47.500496+0000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:57:47.499184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:57:48.763412+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":31361},{"type":"v1","addr":"172.21.15.151:6818","nonce":31361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":31361},{"type":"v1","addr":"172.21.15.151:6824","nonce":31361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":31361},{"type":"v1","addr":"172.21.15.151:6831","nonce":31361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":31361},{"type":"v1","addr":"172.21.15.151:6828","nonce":31361}]},"public_addr":"172.21.15.151:6818/31361","cluster_addr":"172.21.15.151:6824/31361","heartbeat_back_addr":"172.21.15.151:6831/31361","heartbeat_front_addr":"172.21.15.151:6828/31361","state":["exists","up"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":31363},{"type":"v1","addr":"172.21.15.151:6811","nonce":31363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":31363},{"type":"v1","addr":"172.21.15.151:6815","nonce":31363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":31363},{"type":"v1","addr":"172.21.15.151:6823","nonce":31363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":31363},{"type":"v1","addr":"172.21.15.151:6819","nonce":31363}]},"public_addr":"172.21.15.151:6811/31363","cluster_addr":"172.21.15.151:6815/31363","heartbeat_back_addr":"172.21.15.151:6823/31363","heartbeat_front_addr":"172.21.15.151:6819/31363","state":["exists","up"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":31362},{"type":"v1","addr":"172.21.15.151:6810","nonce":31362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":31362},{"type":"v1","addr":"172.21.15.151:6816","nonce":31362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":31362},{"type":"v1","addr":"172.21.15.151:6829","nonce":31362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":31362},{"type":"v1","addr":"172.21.15.151:6825","nonce":31362}]},"public_addr":"172.21.15.151:6810/31362","cluster_addr":"172.21.15.151:6816/31362","heartbeat_back_addr":"172.21.15.151:6829/31362","heartbeat_front_addr":"172.21.15.151:6825/31362","state":["exists","up"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":31360},{"type":"v1","addr":"172.21.15.151:6801","nonce":31360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":31360},{"type":"v1","addr":"172.21.15.151:6803","nonce":31360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":31360},{"type":"v1","addr":"172.21.15.151:6807","nonce":31360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":31360},{"type":"v1","addr":"172.21.15.151:6805","nonce":31360}]},"public_addr":"172.21.15.151:6801/31360","cluster_addr":"172.21.15.151:6803/31360","heartbeat_back_addr":"172.21.15.151:6807/31360","heartbeat_front_addr":"172.21.15.151:6805/31360","state":["exists","up"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":31451},{"type":"v1","addr":"172.21.15.172:6801","nonce":31451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":31451},{"type":"v1","addr":"172.21.15.172:6803","nonce":31451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":31451},{"type":"v1","addr":"172.21.15.172:6807","nonce":31451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":31451},{"type":"v1","addr":"172.21.15.172:6805","nonce":31451}]},"public_addr":"172.21.15.172:6801/31451","cluster_addr":"172.21.15.172:6803/31451","heartbeat_back_addr":"172.21.15.172:6807/31451","heartbeat_front_addr":"172.21.15.172:6805/31451","state":["exists","up"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":31452},{"type":"v1","addr":"172.21.15.172:6811","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":31452},{"type":"v1","addr":"172.21.15.172:6814","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":31452},{"type":"v1","addr":"172.21.15.172:6820","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":31452},{"type":"v1","addr":"172.21.15.172:6817","nonce":31452}]},"public_addr":"172.21.15.172:6811/31452","cluster_addr":"172.21.15.172:6814/31452","heartbeat_back_addr":"172.21.15.172:6820/31452","heartbeat_front_addr":"172.21.15.172:6817/31452","state":["exists","up"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31453},{"type":"v1","addr":"172.21.15.172:6825","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":31453},{"type":"v1","addr":"172.21.15.172:6827","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":31453},{"type":"v1","addr":"172.21.15.172:6831","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":31453},{"type":"v1","addr":"172.21.15.172:6829","nonce":31453}]},"public_addr":"172.21.15.172:6825/31453","cluster_addr":"172.21.15.172:6827/31453","heartbeat_back_addr":"172.21.15.172:6831/31453","heartbeat_front_addr":"172.21.15.172:6829/31453","state":["exists","up"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":31450},{"type":"v1","addr":"172.21.15.172:6809","nonce":31450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":31450},{"type":"v1","addr":"172.21.15.172:6815","nonce":31450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":31450},{"type":"v1","addr":"172.21.15.172:6823","nonce":31450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":31450},{"type":"v1","addr":"172.21.15.172:6821","nonce":31450}]},"public_addr":"172.21.15.172:6809/31450","cluster_addr":"172.21.15.172:6815/31450","heartbeat_back_addr":"172.21.15.172:6823/31450","heartbeat_front_addr":"172.21.15.172:6821/31450","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.403867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.433944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.158817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.278191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.087404+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.063855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.494600+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.286608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 msgr2=0x7f0274062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 0x7f0274062970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f029413a3a0 tx=0x7f028401b000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 msgr2=0x7f02941189a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f028800d890 tx=0x7f0288008040 comp rx=0 tx=0).stop 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 shutdown_connections 2023-09-30T01:57:51.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f02740604c0 0x7f0274062970 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0294139f20 0x7f029414cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02940b0a80 0x7f02941189a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 --2- 172.21.15.151:0/2477407927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f029406ea20 0x7f029413d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 >> 172.21.15.151:0/2477407927 conn(0x7f0294070510 msgr2=0x7f02940af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 shutdown_connections 2023-09-30T01:57:51.192 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.184+0000 7f029359e700 1 -- 172.21.15.151:0/2477407927 wait complete. 2023-09-30T01:57:51.202 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T01:57:51.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:57:51.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 Processor -- start 2023-09-30T01:57:51.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 -- start start 2023-09-30T01:57:51.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe1f00b3b90 0x7fe1f006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe1f00b3f60 0x7fe1f006d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f006c720 con 0x7fe1f006db80 2023-09-30T01:57:51.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f006ca00 con 0x7fe1f00b3b90 2023-09-30T01:57:51.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f61e4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f006b810 con 0x7fe1f00b3f60 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe1f00b3b90 0x7fe1f006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe1f00b3b90 0x7fe1f006d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55424/0 (socket says 172.21.15.151:55424) 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 -- 172.21.15.151:0/3958424441 learned_addr learned my addr 172.21.15.151:0/3958424441 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f49e1700 1 --2- 172.21.15.151:0/3958424441 >> v2:172.21.15.172:3300/0 conn(0x7fe1f00b3f60 0x7fe1f006d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 -- 172.21.15.151:0/3958424441 >> v2:172.21.15.172:3300/0 conn(0x7fe1f00b3f60 msgr2=0x7fe1f006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 --2- 172.21.15.151:0/3958424441 >> v2:172.21.15.172:3300/0 conn(0x7fe1f00b3f60 0x7fe1f006d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 -- 172.21.15.151:0/3958424441 >> v1:172.21.15.172:6789/0 conn(0x7fe1f006db80 legacy=0x7fe1f00ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 -- 172.21.15.151:0/3958424441 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00b0170 con 0x7fe1f00b3b90 2023-09-30T01:57:51.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.300+0000 7fe1f51e2700 1 --2- 172.21.15.151:0/3958424441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3b90 0x7fe1f006d100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1f006bde0 tx=0x7fe1d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629dc1083b6291b6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1effff700 1 -- 172.21.15.151:0/3958424441 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8009e70 con 0x7fe1f00b3b90 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1effff700 1 -- 172.21.15.151:0/3958424441 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d800b860 con 0x7fe1f00b3b90 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1effff700 1 -- 172.21.15.151:0/3958424441 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d800bc20 con 0x7fe1f00b3b90 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- 172.21.15.151:0/3958424441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3b90 msgr2=0x7fe1f006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/3958424441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3b90 0x7fe1f006d100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1f006bde0 tx=0x7fe1d8009250 comp rx=0 tx=0).stop 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- 172.21.15.151:0/3958424441 shutdown_connections 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/3958424441 >> v2:172.21.15.172:3300/0 conn(0x7fe1f00b3f60 0x7fe1f006d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/3958424441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3b90 0x7fe1f006d100 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- 172.21.15.151:0/3958424441 >> 172.21.15.151:0/3958424441 conn(0x7fe1f0063990 msgr2=0x7fe1f0070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- 172.21.15.151:0/3958424441 shutdown_connections 2023-09-30T01:57:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- 172.21.15.151:0/3958424441 wait complete. 2023-09-30T01:57:51.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 Processor -- start 2023-09-30T01:57:51.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.304+0000 7fe1f61e4700 1 -- start start 2023-09-30T01:57:51.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1f006db80 0x7fe1f0150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 0x7fe1f00a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f01506d0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f011c6d0 con 0x7fe1f00b3b90 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00aa8c0 con 0x7fe1f006db80 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55428/0 (socket says 172.21.15.151:55428) 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f51e2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1f006db80 0x7fe1f0150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 -- 172.21.15.151:0/2322840123 learned_addr learned my addr 172.21.15.151:0/2322840123 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f49e1700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 0x7fe1f00a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 -- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1f006db80 msgr2=0x7fe1f0150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1f006db80 0x7fe1f0150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 -- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 msgr2=0x7fe1f00a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 0x7fe1f00a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 -- 172.21.15.151:0/2322840123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d80094f0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f49e1700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 0x7fe1f00a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f59e3700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c4e0 tx=0x7fe1e8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011590 con 0x7fe1f00b3f60 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e80116f0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011ab0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f0150b90 con 0x7fe1f00b3f60 2023-09-30T01:57:51.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f013b100 con 0x7fe1f00b3f60 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.308+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f01508b0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.312+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7fe1e801b3b0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.312+0000 7fe1ed7fa700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 0x7fe1d00629a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.312+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe1e807b910 con 0x7fe1f00b3f60 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.312+0000 7fe1f51e2700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 0x7fe1d00629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.312+0000 7fe1f51e2700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 0x7fe1d00629a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe1f006bde0 tx=0x7fe1d8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.320+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe1f01508b0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.480+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1f01508b0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.480+0000 7fe1ed7fa700 1 -- 172.21.15.151:0/2322840123 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13708 (secure 0 0 0) 0x7fe1f01508b0 con 0x7fe1f00b3f60 2023-09-30T01:57:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:51.486 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":17,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:50.527260+0000","last_up_change":"2023-09-30T01:57:47.500496+0000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:57:47.499184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:57:48.763412+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":31361},{"type":"v1","addr":"172.21.15.151:6818","nonce":31361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":31361},{"type":"v1","addr":"172.21.15.151:6824","nonce":31361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":31361},{"type":"v1","addr":"172.21.15.151:6831","nonce":31361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":31361},{"type":"v1","addr":"172.21.15.151:6828","nonce":31361}]},"public_addr":"172.21.15.151:6818/31361","cluster_addr":"172.21.15.151:6824/31361","heartbeat_back_addr":"172.21.15.151:6831/31361","heartbeat_front_addr":"172.21.15.151:6828/31361","state":["exists","up"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":31363},{"type":"v1","addr":"172.21.15.151:6811","nonce":31363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":31363},{"type":"v1","addr":"172.21.15.151:6815","nonce":31363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":31363},{"type":"v1","addr":"172.21.15.151:6823","nonce":31363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":31363},{"type":"v1","addr":"172.21.15.151:6819","nonce":31363}]},"public_addr":"172.21.15.151:6811/31363","cluster_addr":"172.21.15.151:6815/31363","heartbeat_back_addr":"172.21.15.151:6823/31363","heartbeat_front_addr":"172.21.15.151:6819/31363","state":["exists","up"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":31362},{"type":"v1","addr":"172.21.15.151:6810","nonce":31362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":31362},{"type":"v1","addr":"172.21.15.151:6816","nonce":31362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":31362},{"type":"v1","addr":"172.21.15.151:6829","nonce":31362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":31362},{"type":"v1","addr":"172.21.15.151:6825","nonce":31362}]},"public_addr":"172.21.15.151:6810/31362","cluster_addr":"172.21.15.151:6816/31362","heartbeat_back_addr":"172.21.15.151:6829/31362","heartbeat_front_addr":"172.21.15.151:6825/31362","state":["exists","up"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":31360},{"type":"v1","addr":"172.21.15.151:6801","nonce":31360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":31360},{"type":"v1","addr":"172.21.15.151:6803","nonce":31360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":31360},{"type":"v1","addr":"172.21.15.151:6807","nonce":31360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":31360},{"type":"v1","addr":"172.21.15.151:6805","nonce":31360}]},"public_addr":"172.21.15.151:6801/31360","cluster_addr":"172.21.15.151:6803/31360","heartbeat_back_addr":"172.21.15.151:6807/31360","heartbeat_front_addr":"172.21.15.151:6805/31360","state":["exists","up"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":31451},{"type":"v1","addr":"172.21.15.172:6801","nonce":31451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":31451},{"type":"v1","addr":"172.21.15.172:6803","nonce":31451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":31451},{"type":"v1","addr":"172.21.15.172:6807","nonce":31451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":31451},{"type":"v1","addr":"172.21.15.172:6805","nonce":31451}]},"public_addr":"172.21.15.172:6801/31451","cluster_addr":"172.21.15.172:6803/31451","heartbeat_back_addr":"172.21.15.172:6807/31451","heartbeat_front_addr":"172.21.15.172:6805/31451","state":["exists","up"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":31452},{"type":"v1","addr":"172.21.15.172:6811","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":31452},{"type":"v1","addr":"172.21.15.172:6814","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":31452},{"type":"v1","addr":"172.21.15.172:6820","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":31452},{"type":"v1","addr":"172.21.15.172:6817","nonce":31452}]},"public_addr":"172.21.15.172:6811/31452","cluster_addr":"172.21.15.172:6814/31452","heartbeat_back_addr":"172.21.15.172:6820/31452","heartbeat_front_addr":"172.21.15.172:6817/31452","state":["exists","up"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31453},{"type":"v1","addr":"172.21.15.172:6825","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":31453},{"type":"v1","addr":"172.21.15.172:6827","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":31453},{"type":"v1","addr":"172.21.15.172:6831","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":31453},{"type":"v1","addr":"172.21.15.172:6829","nonce":31453}]},"public_addr":"172.21.15.172:6825/31453","cluster_addr":"172.21.15.172:6827/31453","heartbeat_back_addr":"172.21.15.172:6831/31453","heartbeat_front_addr":"172.21.15.172:6829/31453","state":["exists","up"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":31450},{"type":"v1","addr":"172.21.15.172:6809","nonce":31450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":31450},{"type":"v1","addr":"172.21.15.172:6815","nonce":31450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":31450},{"type":"v1","addr":"172.21.15.172:6823","nonce":31450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":31450},{"type":"v1","addr":"172.21.15.172:6821","nonce":31450}]},"public_addr":"172.21.15.172:6809/31450","cluster_addr":"172.21.15.172:6815/31450","heartbeat_back_addr":"172.21.15.172:6823/31450","heartbeat_front_addr":"172.21.15.172:6821/31450","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.403867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.433944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.158817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.278191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.087404+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.063855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.494600+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.286608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 msgr2=0x7fe1d00629a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 0x7fe1d00629a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe1f006bde0 tx=0x7fe1d8017000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 msgr2=0x7fe1f00a9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c4e0 tx=0x7fe1e8007ed0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 shutdown_connections 2023-09-30T01:57:51.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe1d00604f0 0x7fe1d00629a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1f00b3f60 0x7fe1f00a9070 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1f00b3b90 0x7fe1f00a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 --2- 172.21.15.151:0/2322840123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1f006db80 0x7fe1f0150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 >> 172.21.15.151:0/2322840123 conn(0x7fe1f0063990 msgr2=0x7fe1f00ad290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 shutdown_connections 2023-09-30T01:57:51.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.484+0000 7fe1f61e4700 1 -- 172.21.15.151:0/2322840123 wait complete. 2023-09-30T01:57:51.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T01:57:51.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T01:57:51.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T01:57:51.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T01:57:51.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T01:57:51.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T01:57:51.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T01:57:51.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T01:57:51.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 Processor -- start 2023-09-30T01:57:51.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 -- start start 2023-09-30T01:57:51.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 Processor -- start 2023-09-30T01:57:51.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 -- start start 2023-09-30T01:57:51.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30fc0a7760 0x7f30fc0a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0a31a0 con 0x7f30fc0a9aa0 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0a8ca0 con 0x7f30fc0a9680 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f3100dac700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30fc0a6660 con 0x7f30fc0a7760 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fad9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f30fc0a9aa0 0x7f30fc0aa900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43264/0 (socket says 172.21.15.151:43264) 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fad9d700 1 -- 172.21.15.151:0/3859979568 learned_addr learned my addr 172.21.15.151:0/3859979568 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fa59c700 1 --2- 172.21.15.151:0/3859979568 >> v2:172.21.15.172:3300/0 conn(0x7f30fc0a7760 0x7f30fc0a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30f959a700 1 -- 172.21.15.151:0/3859979568 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556711079 0 0) 0x7f30fc0a31a0 con 0x7f30fc0a9aa0 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30f959a700 1 -- 172.21.15.151:0/3859979568 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e4003570 con 0x7f30fc0a9aa0 2023-09-30T01:57:51.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 0x7f2f0806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fa59c700 1 -- 172.21.15.151:0/3859979568 >> v1:172.21.15.172:6789/0 conn(0x7f30fc0a9680 legacy=0x7f30fc0aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fa59c700 1 -- 172.21.15.151:0/3859979568 >> v1:172.21.15.151:6789/0 conn(0x7f30fc0a9aa0 legacy=0x7f30fc0aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f30fa59c700 1 -- 172.21.15.151:0/3859979568 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0a8f80 con 0x7f30fc0a7760 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f080ad030 0x7f2f08141f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f081424a0 0x7f2f081453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0806d300 con 0x7f2f081424a0 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f08141a80 con 0x7f2f080ad030 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0fdbf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08142900 con 0x7f2f080acc60 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0e35c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f081424a0 0x7f2f081453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0e35c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f081424a0 0x7f2f081453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55432/0 (socket says 172.21.15.151:55432) 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0d35a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f080ad030 0x7f2f08141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0e35c700 1 -- 172.21.15.151:0/3650941205 learned_addr learned my addr 172.21.15.151:0/3650941205 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.644+0000 7f2f0db5b700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 0x7f2f0806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 -- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 msgr2=0x7f2f0806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f30fa59c700 1 --2- 172.21.15.151:0/3859979568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc0a7760 0x7f30fc0a7b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009de0 tx=0x7f30f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f198ed8517c38166 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 0x7f2f0806b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 -- 172.21.15.151:0/3650941205 >> v2:172.21.15.172:3300/0 conn(0x7f2f080ad030 msgr2=0x7f2f08141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/3650941205 >> v2:172.21.15.172:3300/0 conn(0x7f2f080ad030 0x7f2f08141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 -- 172.21.15.151:0/3650941205 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0806b750 con 0x7f2f081424a0 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0db5b700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 0x7f2f0806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f30f959a700 1 -- 172.21.15.151:0/3859979568 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f400bae0 con 0x7f30fc0a7760 2023-09-30T01:57:51.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f30f959a700 1 -- 172.21.15.151:0/3859979568 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f400bc40 con 0x7f30fc0a7760 2023-09-30T01:57:51.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- 172.21.15.151:0/3859979568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc0a7760 msgr2=0x7f30fc0a7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 --2- 172.21.15.151:0/3859979568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc0a7760 0x7f30fc0a7b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009de0 tx=0x7f30f4009780 comp rx=0 tx=0).stop 2023-09-30T01:57:51.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f081453d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f0806a390 tx=0x7f2ef8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354720cdcc599f7f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- 172.21.15.151:0/3859979568 shutdown_connections 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 --2- 172.21.15.151:0/3859979568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc0a7760 0x7f30fc0a7b30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- 172.21.15.151:0/3859979568 >> 172.21.15.151:0/3859979568 conn(0x7f30fc070fe0 msgr2=0x7f30fc0af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- 172.21.15.151:0/3859979568 shutdown_connections 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 Processor -- start 2023-09-30T01:57:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 -- start start 2023-09-30T01:57:51.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- 172.21.15.151:0/3859979568 wait complete. 2023-09-30T01:57:51.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0cb59700 1 -- 172.21.15.151:0/3650941205 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800b720 con 0x7f2f081424a0 2023-09-30T01:57:51.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a4c0ac140 0x7f2a4c0ab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2a4c145200 0x7f2a4c145620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0cb59700 1 -- 172.21.15.151:0/3650941205 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef800b880 con 0x7f2f081424a0 2023-09-30T01:57:51.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c0a6260 con 0x7f2a4c0ac510 2023-09-30T01:57:51.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0ab8a0 con 0x7f2a4c145200 2023-09-30T01:57:51.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a52732700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0a53e0 con 0x7f2a4c0ac140 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 Processor -- start 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f3100dac700 1 -- start start 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 msgr2=0x7f2f081453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2f0fdbf700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f081453d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f0806a390 tx=0x7f2ef8009250 comp rx=0 tx=0).stop 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2a4c145200 0x7f2a4c145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51730700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2a4c0ac140 0x7f2a4c0ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2a4c145200 0x7f2a4c145620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55434/0 (socket says 172.21.15.151:55434) 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 -- 172.21.15.151:0/160486545 learned_addr learned my addr 172.21.15.151:0/160486545 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 -- 172.21.15.151:0/160486545 >> v2:172.21.15.172:3300/0 conn(0x7f2a4c0ac140 msgr2=0x7f2a4c0ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 --2- 172.21.15.151:0/160486545 >> v2:172.21.15.172:3300/0 conn(0x7f2a4c0ac140 0x7f2a4c0ab360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 -- 172.21.15.151:0/160486545 >> v1:172.21.15.172:6789/0 conn(0x7f2a4c0ac510 legacy=0x7f2a4c0a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.648+0000 7f2a51f31700 1 -- 172.21.15.151:0/160486545 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c0a56c0 con 0x7f2a4c145200 2023-09-30T01:57:51.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30fc0a7760 0x7f30fc06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 0x7f30fc0981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc097690 0x7f30fc14add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc098730 con 0x7f30fc0a7760 2023-09-30T01:57:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc098e40 con 0x7f30fc097690 2023-09-30T01:57:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc098910 con 0x7f30fc0a9680 2023-09-30T01:57:51.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a51f31700 1 --2- 172.21.15.151:0/160486545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c145200 0x7f2a4c145620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c06a170 tx=0x7f2a44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca48beb3b2bbcac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 0x7f30fc0981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 0x7f30fc0981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44464/0 (socket says 172.21.15.151:44464) 2023-09-30T01:57:51.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 -- 172.21.15.151:0/3888674860 learned_addr learned my addr 172.21.15.151:0/3888674860 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30fad9d700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc097690 0x7f30fc14add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/3650941205 shutdown_connections 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f081453d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 --2- 172.21.15.151:0/3650941205 >> v2:172.21.15.172:3300/0 conn(0x7f2f080ad030 0x7f2f08141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 --2- 172.21.15.151:0/3650941205 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080acc60 0x7f2f0806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/3650941205 >> 172.21.15.151:0/3650941205 conn(0x7f2f080708b0 msgr2=0x7f2f08070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30fa59c700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30fc0a7760 0x7f30fc06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 -- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc097690 msgr2=0x7f30fc14add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc097690 0x7f30fc14add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 -- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30fc0a7760 msgr2=0x7f30fc06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30fc0a7760 0x7f30fc06e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4009a20 con 0x7f30fc0a9680 2023-09-30T01:57:51.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30fad9d700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30fc097690 0x7f30fc14add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30f9d9b700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 0x7f30fc0981f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c810 tx=0x7f30f0007f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/3650941205 shutdown_connections 2023-09-30T01:57:51.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a43fff700 1 -- 172.21.15.151:0/160486545 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400e070 con 0x7f2a4c145200 2023-09-30T01:57:51.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a43fff700 1 -- 172.21.15.151:0/160486545 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4400b9e0 con 0x7f2a4c145200 2023-09-30T01:57:51.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a43fff700 1 -- 172.21.15.151:0/160486545 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a440092e0 con 0x7f2a4c145200 2023-09-30T01:57:51.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 -- 172.21.15.151:0/160486545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c145200 msgr2=0x7f2a4c145620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 --2- 172.21.15.151:0/160486545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c145200 0x7f2a4c145620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c06a170 tx=0x7f2a44009840 comp rx=0 tx=0).stop 2023-09-30T01:57:51.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/3650941205 wait complete. 2023-09-30T01:57:51.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 -- 172.21.15.151:0/160486545 shutdown_connections 2023-09-30T01:57:51.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 --2- 172.21.15.151:0/160486545 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c145200 0x7f2a4c145620 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 --2- 172.21.15.151:0/160486545 >> v2:172.21.15.172:3300/0 conn(0x7f2a4c0ac140 0x7f2a4c0ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 -- 172.21.15.151:0/160486545 >> 172.21.15.151:0/160486545 conn(0x7f2a4c070510 msgr2=0x7f2a4c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0012630 con 0x7f30fc0a9680 2023-09-30T01:57:51.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f00127d0 con 0x7f30fc0a9680 2023-09-30T01:57:51.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 -- 172.21.15.151:0/160486545 shutdown_connections 2023-09-30T01:57:51.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 -- 172.21.15.151:0/160486545 wait complete. 2023-09-30T01:57:51.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 Processor -- start 2023-09-30T01:57:51.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30fc14b370 con 0x7f30fc0a9680 2023-09-30T01:57:51.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc14b8c0 con 0x7f30fc0a9680 2023-09-30T01:57:51.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0012b90 con 0x7f30fc0a9680 2023-09-30T01:57:51.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 -- start start 2023-09-30T01:57:51.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 Processor -- start 2023-09-30T01:57:51.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2a52732700 1 Processor -- start 2023-09-30T01:57:51.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fc52af34700 1 Processor -- start 2023-09-30T01:57:51.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fc52af34700 1 -- start start 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd29c0a92b0 0x7fd29c0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd29c145090 0x7fd29c0ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c1457f0 con 0x7fd29c0a92b0 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c145510 con 0x7fd29c145090 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd2a17a0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0abbe0 con 0x7fd29c0ab740 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd29b7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd29b7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44468/0 (socket says 172.21.15.151:44468) 2023-09-30T01:57:51.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd29b7fe700 1 -- 172.21.15.151:0/2087271020 learned_addr learned my addr 172.21.15.151:0/2087271020 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7fd29a7fc700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.172:3300/0 conn(0x7fd29c145090 0x7fd29c0ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.652+0000 7f2f0fdbf700 1 -- start start 2023-09-30T01:57:51.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29affd700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.151:3300/0 conn(0x7fd29c0a92b0 0x7fd29c0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- start start 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f080acc60 0x7f2f080add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080ad030 0x7f2f08146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f080b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080ae4c0 con 0x7f2f081424a0 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080b0160 con 0x7f2f080acc60 2023-09-30T01:57:51.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080b0ac0 con 0x7f2f080ad030 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 -- 172.21.15.151:0/2087271020 >> v2:172.21.15.172:3300/0 conn(0x7fd29c145090 msgr2=0x7fd29c0ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.172:3300/0 conn(0x7fd29c145090 0x7fd29c0ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 -- 172.21.15.151:0/2087271020 >> v2:172.21.15.151:3300/0 conn(0x7fd29c0a92b0 msgr2=0x7fd29c0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.151:3300/0 conn(0x7fd29c0a92b0 0x7fd29c0a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 -- 172.21.15.151:0/2087271020 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0ac0f0 con 0x7fd29c0ab740 2023-09-30T01:57:51.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f080b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc52af34700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc5240b01c0 0x7fc5240a9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc52af34700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc524142140 0x7fc5240aa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc52af34700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240a9cb0 con 0x7fc524141d00 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc52af34700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52413f300 con 0x7fc524142140 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc52af34700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52406f750 con 0x7fc5240b01c0 2023-09-30T01:57:51.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f080b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55456/0 (socket says 172.21.15.151:55456) 2023-09-30T01:57:51.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 -- 172.21.15.151:0/2593063560 learned_addr learned my addr 172.21.15.151:0/2593063560 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2087271020 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0a8cc0 tx=0x7fd294009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6ae8d5f80cf7445 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0db5b700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f080acc60 0x7f2f080add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 -- 172.21.15.151:0/2593063560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080ad030 msgr2=0x7f2f08146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f080ad030 0x7f2f08146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 -- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f080acc60 msgr2=0x7f2f080add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f080acc60 0x7f2f080add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a4c0ac140 0x7f2a4c150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 Processor -- start 2023-09-30T01:57:51.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc528cd0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc5240b01c0 0x7fc5240a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc523fff700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc524141d00 0x7fc5240aa1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41274/0 (socket says 172.21.15.151:41274) 2023-09-30T01:57:51.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc523fff700 1 -- 172.21.15.151:0/2982527352 learned_addr learned my addr 172.21.15.151:0/2982527352 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd299ffb700 1 -- 172.21.15.151:0/2087271020 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400e070 con 0x7fd29c0ab740 2023-09-30T01:57:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd299ffb700 1 -- 172.21.15.151:0/2087271020 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400b970 con 0x7fd29c0ab740 2023-09-30T01:57:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd299ffb700 1 -- 172.21.15.151:0/2087271020 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940092e0 con 0x7fd29c0ab740 2023-09-30T01:57:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef80094f0 con 0x7f2f081424a0 2023-09-30T01:57:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0e35c700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f080b0580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009ae0 tx=0x7f2ef8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 0x7f2a4c1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a4c145200 0x7f2a4c11a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c11a010 con 0x7f2a4c0ac510 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c151520 con 0x7f2a4c145200 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c11a210 con 0x7f2a4c0ac140 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 0x7f2a4c1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 0x7f2a4c1519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55474/0 (socket says 172.21.15.151:55474) 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 -- 172.21.15.151:0/334877108 learned_addr learned my addr 172.21.15.151:0/334877108 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 -- start start 2023-09-30T01:57:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a51f31700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a4c145200 0x7f2a4c11a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a51730700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a4c0ac140 0x7f2a4c150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 -- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a4c0ac140 msgr2=0x7f2a4c150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a4c0ac140 0x7f2a4c150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 -- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a4c145200 msgr2=0x7f2a4c11a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a4c145200 0x7f2a4c11a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44009ae0 con 0x7f2a4c0ac510 2023-09-30T01:57:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc5294d1700 1 --2- 172.21.15.151:0/2982527352 >> v2:172.21.15.151:3300/0 conn(0x7fc524142140 0x7fc5240aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8002f70 con 0x7f2f081424a0 2023-09-30T01:57:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2087271020 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 msgr2=0x7fd29c0a6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2087271020 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0a8cc0 tx=0x7fd294009840 comp rx=0 tx=0).stop 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f30f001b400 con 0x7f30fc0a9680 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2087271020 shutdown_connections 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2087271020 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c0a6d70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.172:3300/0 conn(0x7fd29c145090 0x7fd29c0ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2087271020 >> v2:172.21.15.151:3300/0 conn(0x7fd29c0a92b0 0x7fd29c0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2087271020 >> 172.21.15.151:0/2087271020 conn(0x7fd29c0af880 msgr2=0x7fd29c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef801ae30 con 0x7f2f081424a0 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8011860 con 0x7f2f081424a0 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f081487e0 con 0x7f2f081424a0 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f080aeab0 con 0x7f2f081424a0 2023-09-30T01:57:51.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc5237fe700 1 -- 172.21.15.151:0/2982527352 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796882163 0 0) 0x7fc5240a9cb0 con 0x7fc524141d00 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc5237fe700 1 -- 172.21.15.151:0/2982527352 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c003570 con 0x7fc524141d00 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a50f2f700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 0x7f2a4c1519b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00b610 tx=0x7f2a3c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00f510 con 0x7f2a4c0ac510 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c00f670 con 0x7f2a4c0ac510 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00fa30 con 0x7f2a4c0ac510 2023-09-30T01:57:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0360 0x7ff9840a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840a3f70 con 0x7ff9840aaf60 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840a4250 con 0x7ff9840aff90 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff988938700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840a3d90 con 0x7ff9840b0360 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff982d9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff9840aaf60 0x7ff9840a61b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43280/0 (socket says 172.21.15.151:43280) 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff982d9d700 1 -- 172.21.15.151:0/312648699 learned_addr learned my addr 172.21.15.151:0/312648699 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322933536 0 0) 0x7ff9840a3f70 con 0x7ff9840aaf60 2023-09-30T01:57:51.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c003570 con 0x7ff9840aaf60 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30fc0008d0 con 0x7f30fc0a9680 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f30dc060620 0x7f30dc062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f30f007c300 con 0x7f30fc0a9680 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7f30dc066880 0x7f30dc068c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30dc069340 con 0x7f30dc066880 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30f007e7e0 con 0x7f30fc0a9680 2023-09-30T01:57:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30fa59c700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f30dc060620 0x7f30dc062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f30fad9d700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7f30dc066880 0x7f30dc068c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc528cd0700 1 -- 172.21.15.151:0/2982527352 >> v2:172.21.15.151:3300/0 conn(0x7fc524142140 msgr2=0x7fc5240aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc528cd0700 1 --2- 172.21.15.151:0/2982527352 >> v2:172.21.15.151:3300/0 conn(0x7fc524142140 0x7fc5240aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc528cd0700 1 -- 172.21.15.151:0/2982527352 >> v1:172.21.15.172:6789/0 conn(0x7fc524141d00 legacy=0x7fc5240aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7fc528cd0700 1 -- 172.21.15.151:0/2982527352 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240b0590 con 0x7fc5240b01c0 2023-09-30T01:57:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c06dd20 con 0x7f2a4c0ac510 2023-09-30T01:57:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c117d60 con 0x7f2a4c0ac510 2023-09-30T01:57:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2087271020 shutdown_connections 2023-09-30T01:57:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f30fad9d700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7f30dc066880 0x7f30dc068c90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc528cd0700 1 --2- 172.21.15.151:0/2982527352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5240b01c0 0x7fc5240a9770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc52406abe0 tx=0x7fc51c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac1c498e39d7c53 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534620148 0 0) 0x7ff96c003570 con 0x7ff9840aaf60 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.656+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0360 msgr2=0x7ff9840a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff981d9b700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0360 0x7ff9840a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0360 0x7ff9840a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 >> v1:172.21.15.172:6789/0 conn(0x7ff9840aff90 legacy=0x7ff98413e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc5237fe700 1 -- 172.21.15.151:0/2982527352 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00e070 con 0x7fc5240b01c0 2023-09-30T01:57:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840a4530 con 0x7ff9840aaf60 2023-09-30T01:57:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7ff97c0017f0 con 0x7ff9840aaf60 2023-09-30T01:57:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 >> v1:172.21.15.151:6789/0 conn(0x7ff9840aaf60 legacy=0x7ff9840a61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c003a90 0x7ff96c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc5237fe700 1 -- 172.21.15.151:0/2982527352 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c00bab0 con 0x7fc5240b01c0 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc5237fe700 1 -- 172.21.15.151:0/2982527352 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00be70 con 0x7fc5240b01c0 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff96c004810 0x7ff96c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff96c0052e0 0x7ff96c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840a3d90 con 0x7ff96c003a90 2023-09-30T01:57:51.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c00c230 con 0x7ff96c004810 2023-09-30T01:57:51.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c00c3f0 con 0x7ff96c0052e0 2023-09-30T01:57:51.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2087271020 wait complete. 2023-09-30T01:57:51.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 Processor -- start 2023-09-30T01:57:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff96c0052e0 0x7ff96c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff981d9b700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0360 0x7ff9840a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== osd.1 v2:172.21.15.151:6809/31363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f30dc069340 con 0x7f30dc066880 2023-09-30T01:57:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f30fa59c700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f30dc060620 0x7f30dc062ad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30f40099a0 tx=0x7f30f401a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- 172.21.15.151:0/2982527352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5240b01c0 msgr2=0x7fc5240a9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 --2- 172.21.15.151:0/2982527352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5240b01c0 0x7fc5240a9770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc52406abe0 tx=0x7fc51c009780 comp rx=0 tx=0).stop 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- 172.21.15.151:0/2982527352 shutdown_connections 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 --2- 172.21.15.151:0/2982527352 >> v2:172.21.15.151:3300/0 conn(0x7fc524142140 0x7fc5240aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 --2- 172.21.15.151:0/2982527352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc5240b01c0 0x7fc5240a9770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- 172.21.15.151:0/2982527352 >> 172.21.15.151:0/2982527352 conn(0x7fc524070fe0 msgr2=0x7fc5240af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 Processor -- start 2023-09-30T01:57:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f080670a0 con 0x7f2f081424a0 2023-09-30T01:57:51.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 -- start start 2023-09-30T01:57:51.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a4c0008d0 con 0x7f2a4c0ac510 2023-09-30T01:57:51.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 -- 172.21.15.151:0/312648699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff96c004810 msgr2=0x7ff96c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:57:51.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff96c004810 0x7ff96c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 -- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c003a90 msgr2=0x7ff96c003e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:51.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c003a90 0x7ff96c003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 -- 172.21.15.151:0/312648699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840a4530 con 0x7ff96c0052e0 2023-09-30T01:57:51.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- 172.21.15.151:0/2982527352 shutdown_connections 2023-09-30T01:57:51.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- 172.21.15.151:0/2982527352 wait complete. 2023-09-30T01:57:51.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff982d9d700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff96c0052e0 0x7ff96c00ba60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00aaa0 tx=0x7ff97c00a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea6a90464aaec5d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- start start 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd29c0a92b0 0x7fd29c141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c145090 0x7fd29c1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c150a60 con 0x7fd29c0a92b0 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0a4db0 con 0x7fd29c145090 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd2a17a0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0a4f70 con 0x7fd29c0ab740 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44512/0 (socket says 172.21.15.151:44512) 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 -- 172.21.15.151:0/2141000685 learned_addr learned my addr 172.21.15.151:0/2141000685 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c145090 0x7fd29c1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 Processor -- start 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2ef8011c50 con 0x7f2f081424a0 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2efdffb700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2eec0606d0 0x7f2eec062b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2ef807dd50 con 0x7f2f081424a0 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2efdffb700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f2eec066aa0 0x7f2eec068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2eec069560 con 0x7f2eec066aa0 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3df406b180 0x7f3df40ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 0x7f3df413d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df413cbf0 con 0x7f3df4139610 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df406c830 con 0x7f3df406b180 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3dfac0a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df406b550 con 0x7f3df406d1a0 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3df9c08700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3df406b180 0x7f3df40ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3df9c08700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3df406b180 0x7f3df40ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50744/0 (socket says 172.21.15.151:50744) 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f3df9c08700 1 -- 172.21.15.151:0/2239616179 learned_addr learned my addr 172.21.15.151:0/2239616179 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2a3c024430 con 0x7f2a4c0ac510 2023-09-30T01:57:51.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fc52af34700 1 -- start start 2023-09-30T01:57:51.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2f0d35a700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f2eec066aa0 0x7f2eec068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 -- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c145090 msgr2=0x7fd29c1514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd29c145090 0x7fd29c1514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00c590 con 0x7ff96c0052e0 2023-09-30T01:57:51.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 -- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd29c0a92b0 msgr2=0x7fd29c141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd29c0a92b0 0x7fd29c141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7fd29a7fc700 1 -- 172.21.15.151:0/2141000685 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009ae0 con 0x7fd29c0ab740 2023-09-30T01:57:51.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00c6f0 con 0x7ff96c0052e0 2023-09-30T01:57:51.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7ff98159a700 1 -- 172.21.15.151:0/312648699 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00cab0 con 0x7ff96c0052e0 2023-09-30T01:57:51.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd29a7fc700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c150f80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd29000ec50 tx=0x7fd2900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2f0db5b700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2eec0606d0 0x7f2eec062b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9407700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 0x7f3df413d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df8c06700 1 -- 172.21.15.151:0/2239616179 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41069374 0 0) 0x7f3df413cbf0 con 0x7f3df4139610 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df8c06700 1 -- 172.21.15.151:0/2239616179 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0003490 con 0x7f3df4139610 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9c08700 1 -- 172.21.15.151:0/2239616179 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 msgr2=0x7f3df413d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9c08700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 0x7f3df413d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9c08700 1 -- 172.21.15.151:0/2239616179 >> v1:172.21.15.172:6789/0 conn(0x7f3df4139610 legacy=0x7f3df4139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9c08700 1 -- 172.21.15.151:0/2239616179 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df413c8c0 con 0x7f3df406b180 2023-09-30T01:57:51.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9407700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 0x7f3df413d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.660+0000 7f2a417fa700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a300606a0 0x7f2a30062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2a3c07c9d0 con 0x7f2a4c0ac510 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a417fa700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f2a30066a70 0x7f2a30068e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a30069530 con 0x7f2a30066a70 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a3c07cd00 con 0x7f2a4c0ac510 2023-09-30T01:57:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29000c850 con 0x7fd29c0ab740 2023-09-30T01:57:51.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 0x7fc5240a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc524141d00 0x7fc5240a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc524142140 0x7fc5241443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524143f60 con 0x7fc524141d00 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240a3490 con 0x7fc524142140 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240a3600 con 0x7fc5240b01c0 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 0x7fc5240a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 0x7fc5240a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44530/0 (socket says 172.21.15.151:44530) 2023-09-30T01:57:51.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 -- 172.21.15.151:0/2544735421 learned_addr learned my addr 172.21.15.151:0/2544735421 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ef804e510 con 0x7f2f081424a0 2023-09-30T01:57:51.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a51730700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a300606a0 0x7f2a30062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a51f31700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f2a30066a70 0x7f2a30068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2f0d35a700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f2eec066aa0 0x7f2eec068eb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff96c0052e0 msgr2=0x7ff96c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 --2- 172.21.15.151:0/312648699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff96c0052e0 0x7ff96c00ba60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00aaa0 tx=0x7ff97c00a0f0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff982d9d700 1 -- 172.21.15.151:0/312648699 reap_dead start 2023-09-30T01:57:51.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc523fff700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc524141d00 0x7fc5240a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc5294d1700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc524142140 0x7fc5241443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- 172.21.15.151:0/312648699 shutdown_connections 2023-09-30T01:57:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- 172.21.15.151:0/312648699 >> 172.21.15.151:0/312648699 conn(0x7ff984070fe0 msgr2=0x7ff9840af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29000c9b0 con 0x7fd29c0ab740 2023-09-30T01:57:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29000cd70 con 0x7fd29c0ab740 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 -- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc524142140 msgr2=0x7fc5241443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc524142140 0x7fc5241443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 -- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc524141d00 msgr2=0x7fc5240a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc524141d00 0x7fc5240a5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 -- 172.21.15.151:0/2544735421 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009a20 con 0x7fc5240b01c0 2023-09-30T01:57:51.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2f0db5b700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2eec0606d0 0x7f2eec062b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2f04006f00 tx=0x7f2f04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- 172.21.15.151:0/312648699 shutdown_connections 2023-09-30T01:57:51.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c0a3840 con 0x7fd29c0ab740 2023-09-30T01:57:51.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c0a3d40 con 0x7fd29c0ab740 2023-09-30T01:57:51.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- 172.21.15.151:0/312648699 wait complete. 2023-09-30T01:57:51.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc528cd0700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 0x7fc5240a4300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00b610 tx=0x7fc51c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df9c08700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406b180 0x7f3df40ac440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009320 tx=0x7f3df0009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c42249d43db42cf4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 Processor -- start 2023-09-30T01:57:51.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 -- start start 2023-09-30T01:57:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f061814a5b0 0x7f061814c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0618145ad0 con 0x7f06180acc20 2023-09-30T01:57:51.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180acff0 con 0x7f061806b280 2023-09-30T01:57:51.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061e2f6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061806bdf0 con 0x7f061814a5b0 2023-09-30T01:57:51.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f0617fff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f06180acc20 0x7f0618145010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43296/0 (socket says 172.21.15.151:43296) 2023-09-30T01:57:51.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f0617fff700 1 -- 172.21.15.151:0/1769355370 learned_addr learned my addr 172.21.15.151:0/1769355370 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f061c893700 1 --2- 172.21.15.151:0/1769355370 >> v2:172.21.15.172:3300/0 conn(0x7f061814a5b0 0x7f061814c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df8c06700 1 -- 172.21.15.151:0/2239616179 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0009810 con 0x7f3df406b180 2023-09-30T01:57:51.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df8c06700 1 -- 172.21.15.151:0/2239616179 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df000b860 con 0x7f3df406b180 2023-09-30T01:57:51.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3df8c06700 1 -- 172.21.15.151:0/2239616179 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000bc00 con 0x7f3df406b180 2023-09-30T01:57:51.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a51730700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a300606a0 0x7f2a30062b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009d10 tx=0x7f2a480091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2a51f31700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f2a30066a70 0x7f2a30068e80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== osd.4 v2:172.21.15.172:6800/31451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2eec069560 con 0x7f2eec066aa0 2023-09-30T01:57:51.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c012c00 con 0x7fc5240b01c0 2023-09-30T01:57:51.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c012d80 con 0x7fc5240b01c0 2023-09-30T01:57:51.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5240a4980 con 0x7fc5240b01c0 2023-09-30T01:57:51.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524144b10 con 0x7fc5240b01c0 2023-09-30T01:57:51.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0276c0 con 0x7fc5240b01c0 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 Processor -- start 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- start start 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c0052e0 0x7ff984095c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff984150be0 0x7ff984151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff984153d30 0x7ff984154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984097f60 con 0x7ff96c0052e0 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98406e250 con 0x7ff984150be0 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff988938700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff984098380 con 0x7ff984153d30 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f0616ffd700 1 -- 172.21.15.151:0/1769355370 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785972002 0 0) 0x7f06180acff0 con 0x7f061806b280 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f0616ffd700 1 -- 172.21.15.151:0/1769355370 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0600003570 con 0x7f061806b280 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061c893700 1 -- 172.21.15.151:0/1769355370 >> v1:172.21.15.172:6789/0 conn(0x7f061806b280 legacy=0x7f061806b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061c893700 1 -- 172.21.15.151:0/1769355370 >> v1:172.21.15.151:6789/0 conn(0x7f06180acc20 legacy=0x7f0618145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061c893700 1 -- 172.21.15.151:0/1769355370 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061806d290 con 0x7f061814a5b0 2023-09-30T01:57:51.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd290013680 con 0x7fd29c0ab740 2023-09-30T01:57:51.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fd282ffd700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd2880605d0 0x7fd288062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd29007ca50 con 0x7fd29c0ab740 2023-09-30T01:57:51.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fd29affd700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd2880605d0 0x7fd288062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2239616179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406b180 msgr2=0x7f3df40ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7f3dfac0a700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406b180 0x7f3df40ac440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009320 tx=0x7f3df0009ae0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2239616179 shutdown_connections 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406d1a0 0x7f3df413d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 --2- 172.21.15.151:0/2239616179 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406b180 0x7f3df40ac440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2239616179 >> 172.21.15.151:0/2239616179 conn(0x7f3df40687a0 msgr2=0x7f3df4070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2239616179 shutdown_connections 2023-09-30T01:57:51.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2239616179 wait complete. 2023-09-30T01:57:51.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== osd.6 v2:172.21.15.172:6824/31453 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2a30069530 con 0x7f2a30066a70 2023-09-30T01:57:51.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff98259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c0052e0 0x7ff984095c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff98259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c0052e0 0x7ff984095c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55506/0 (socket says 172.21.15.151:55506) 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.664+0000 7ff98259c700 1 -- 172.21.15.151:0/255300976 learned_addr learned my addr 172.21.15.151:0/255300976 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff984153d30 0x7ff984154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff981d9b700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff984150be0 0x7ff984151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 -- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff984150be0 msgr2=0x7ff984151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff984150be0 0x7ff984151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 -- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c0052e0 msgr2=0x7ff984095c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff96c0052e0 0x7ff984095c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984151550 con 0x7ff984153d30 2023-09-30T01:57:51.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7ff982d9d700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff984153d30 0x7ff984154180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff97c015000 tx=0x7ff97c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc51c024070 con 0x7fc5240b01c0 2023-09-30T01:57:51.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061c893700 1 --2- 172.21.15.151:0/1769355370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061814a5b0 0x7f061814c990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06080098e0 tx=0x7f06080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f8d7a6d84b34c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fc520ff9700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc510060650 0x7fc510062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fd29affd700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd2880605d0 0x7fd288062a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009b10 tx=0x7fd28c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc51c07e9a0 con 0x7fc5240b01c0 2023-09-30T01:57:51.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fc523fff700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc510060650 0x7fc510062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 Processor -- start 2023-09-30T01:57:51.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f0616ffd700 1 -- 172.21.15.151:0/1769355370 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060800b640 con 0x7f061814a5b0 2023-09-30T01:57:51.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f0616ffd700 1 -- 172.21.15.151:0/1769355370 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060800b7c0 con 0x7f061814a5b0 2023-09-30T01:57:51.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f0616ffd700 1 -- 172.21.15.151:0/1769355370 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060800bb80 con 0x7f061814a5b0 2023-09-30T01:57:51.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- start start 2023-09-30T01:57:51.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7fc523fff700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc510060650 0x7fc510062b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc524145960 tx=0x7fc518009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061e2f6700 1 -- 172.21.15.151:0/1769355370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061814a5b0 msgr2=0x7f061814c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f061e2f6700 1 --2- 172.21.15.151:0/1769355370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061814a5b0 0x7f061814c990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06080098e0 tx=0x7f06080091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 0x7f3df4142e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406d1a0 0x7f3df4143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3df4139610 0x7f3df414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df4144200 con 0x7f3df4139610 2023-09-30T01:57:51.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df4117a50 con 0x7f3df406d1a0 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfac0a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df4117c50 con 0x7f3df406b180 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3df9c08700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 0x7f3df4142e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3df9c08700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 0x7f3df4142e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44550/0 (socket says 172.21.15.151:44550) 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3df9c08700 1 -- 172.21.15.151:0/2705159916 learned_addr learned my addr 172.21.15.151:0/2705159916 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3dfa409700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3df4139610 0x7f3df414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.668+0000 7f3df9407700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406d1a0 0x7f3df4143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 -- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406d1a0 msgr2=0x7f3df4143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df406d1a0 0x7f3df4143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 -- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3df4139610 msgr2=0x7f3df414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3df4139610 0x7f3df414c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0009d80 con 0x7f3df406b180 2023-09-30T01:57:51.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0017b0 con 0x7ff984153d30 2023-09-30T01:57:51.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f061e2f6700 1 -- 172.21.15.151:0/1769355370 shutdown_connections 2023-09-30T01:57:51.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f061e2f6700 1 --2- 172.21.15.151:0/1769355370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061814a5b0 0x7f061814c990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f061e2f6700 1 -- 172.21.15.151:0/1769355370 >> 172.21.15.151:0/1769355370 conn(0x7f0618070510 msgr2=0x7f0618070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3dfa409700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3df4139610 0x7f3df414c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:51.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7fd29c0a4020 0x7fd29c13e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3df9c08700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 0x7f3df4142e60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009b20 tx=0x7f3df0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00ca70 con 0x7ff984153d30 2023-09-30T01:57:51.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c01f440 con 0x7ff984153d30 2023-09-30T01:57:51.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 --> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd29c150bd0 con 0x7fd29c0a4020 2023-09-30T01:57:51.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff988938700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff984096140 con 0x7ff984153d30 2023-09-30T01:57:51.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff988938700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff984099b60 con 0x7ff984153d30 2023-09-30T01:57:51.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7fd29c0a4020 0x7fd29c13e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7fd29b7fe700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7fd29c0a4020 0x7fd29c13e420 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000bbe0 con 0x7f3df406b180 2023-09-30T01:57:51.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df001f070 con 0x7f3df406b180 2023-09-30T01:57:51.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df00266d0 con 0x7f3df406b180 2023-09-30T01:57:51.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df4142010 con 0x7f3df406b180 2023-09-30T01:57:51.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3dfac0a700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df414cf60 con 0x7f3df406b180 2023-09-30T01:57:51.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff988938700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff984150820 con 0x7ff984153d30 2023-09-30T01:57:51.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff97c025020 con 0x7ff984153d30 2023-09-30T01:57:51.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff964060620 0x7ff964062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff97c07ed40 con 0x7ff984153d30 2023-09-30T01:57:51.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== osd.3 v2:172.21.15.151:6800/31360 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fd29c150bd0 con 0x7fd29c0a4020 2023-09-30T01:57:51.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7ff9640669f0 0x7ff964068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9640694b0 con 0x7ff9640669f0 2023-09-30T01:57:51.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3df40008d0 con 0x7f3df406b180 2023-09-30T01:57:51.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff98259c700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff964060620 0x7ff964062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff97c00a0f0 con 0x7ff984153d30 2023-09-30T01:57:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.672+0000 7ff981d9b700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7ff9640669f0 0x7ff964068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- 172.21.15.151:0/1769355370 shutdown_connections 2023-09-30T01:57:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3df001a430 con 0x7f3df406b180 2023-09-30T01:57:51.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7ff98259c700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff964060620 0x7ff964062ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff974003be0 tx=0x7ff974007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7ff981d9b700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7ff9640669f0 0x7ff964068e00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3de0060aa0 0x7f3de0062f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3df0021070 con 0x7f3df406b180 2023-09-30T01:57:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f3de0066e70 0x7f3de0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3df9407700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3de0060aa0 0x7f3de0062f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3de0069930 con 0x7f3de0066e70 2023-09-30T01:57:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- 172.21.15.151:0/1769355370 wait complete. 2023-09-30T01:57:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3df007e710 con 0x7f3df406b180 2023-09-30T01:57:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3dfa409700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f3de0066e70 0x7f3de0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3df9407700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3de0060aa0 0x7f3de0062f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3df4141d90 tx=0x7f3de4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3dfa409700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f3de0066e70 0x7f3de0069280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 Processor -- start 2023-09-30T01:57:51.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- start start 2023-09-30T01:57:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== osd.5 v2:172.21.15.172:6810/31452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3de0069930 con 0x7f3de0066e70 2023-09-30T01:57:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 0x7f06181423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f06180acc20 0x7f0618139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f061814a5b0 0x7f061813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618138f00 con 0x7f061814a5b0 2023-09-30T01:57:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06181429f0 con 0x7f061806b280 2023-09-30T01:57:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f061e2f6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061813e770 con 0x7f06180acc20 2023-09-30T01:57:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== osd.7 v2:172.21.15.172:6808/31450 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff9640694b0 con 0x7ff9640669f0 2023-09-30T01:57:51.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f0617fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 0x7f06181423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f0617fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 0x7f06181423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50776/0 (socket says 172.21.15.151:50776) 2023-09-30T01:57:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.676+0000 7f0617fff700 1 -- 172.21.15.151:0/4064607366 learned_addr learned my addr 172.21.15.151:0/4064607366 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f06177fe700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f06180acc20 0x7f0618139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 -- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f06180acc20 msgr2=0x7f0618139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f06180acc20 0x7f0618139390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 -- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f061814a5b0 msgr2=0x7f061813f880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:57:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f061814a5b0 0x7f061813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608009420 con 0x7f061806b280 2023-09-30T01:57:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f0617fff700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 0x7f06181423d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f061000cc80 tx=0x7f0610001a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000a960 con 0x7f061806b280 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061813c890 con 0x7f061806b280 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061813f550 con 0x7f061806b280 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061000aae0 con 0x7f061806b280 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000aea0 con 0x7f061806b280 2023-09-30T01:57:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f06180670a0 con 0x7f061806b280 2023-09-30T01:57:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7fc52af34700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7fc5040015f0 0x7fc504003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 --> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc504006ae0 con 0x7fc5040015f0 2023-09-30T01:57:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.680+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0610007300 con 0x7f061806b280 2023-09-30T01:57:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f05fffff700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f06040605d0 0x7f0604062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f061007d480 con 0x7f061806b280 2023-09-30T01:57:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f06177fe700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f06040605d0 0x7f0604062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7fc5294d1700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7fc5040015f0 0x7fc504003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f05fffff700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7f06040669a0 0x7f0604068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0604069460 con 0x7f06040669a0 2023-09-30T01:57:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f061007d7b0 con 0x7f061806b280 2023-09-30T01:57:51.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7fc5294d1700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7fc5040015f0 0x7fc504003aa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7f06177fe700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f06040605d0 0x7f0604062a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0618141bb0 tx=0x7f060c007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.684+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== osd.2 v2:172.21.15.151:6808/31362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc504006ae0 con 0x7fc5040015f0 2023-09-30T01:57:51.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 --> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f30fc0008d0 con 0x7f30dc066880 2023-09-30T01:57:51.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 --> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a4c118010 con 0x7f2a30066a70 2023-09-30T01:57:51.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f061c893700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7f06040669a0 0x7f0604068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f30ea7fc700 1 -- 172.21.15.151:0/3888674860 <== osd.1 v2:172.21.15.151:6809/31363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f30fc0008d0 con 0x7f30dc066880 2023-09-30T01:57:51.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f2a417fa700 1 -- 172.21.15.151:0/334877108 <== osd.6 v2:172.21.15.172:6824/31453 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2a4c118010 con 0x7f2a30066a70 2023-09-30T01:57:51.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f061c893700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7f06040669a0 0x7f0604068db0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7f30dc066880 msgr2=0x7f30dc068c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7f30dc066880 0x7f30dc068c90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f30dc060620 msgr2=0x7f30dc062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f30dc060620 0x7f30dc062ad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30f40099a0 tx=0x7f30f401a000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 msgr2=0x7f30fc0981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 --2- 172.21.15.151:0/3888674860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30fc0a9680 0x7f30fc0981f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c810 tx=0x7f30f0007f30 comp rx=0 tx=0).stop 2023-09-30T01:57:51.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f30fad9d700 1 -- 172.21.15.151:0/3888674860 reap_dead start 2023-09-30T01:57:51.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 shutdown_connections 2023-09-30T01:57:51.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 >> 172.21.15.151:0/3888674860 conn(0x7f30fc070fe0 msgr2=0x7f30fc0af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 shutdown_connections 2023-09-30T01:57:51.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f3100dac700 1 -- 172.21.15.151:0/3888674860 wait complete. 2023-09-30T01:57:51.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.688+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== osd.0 v2:172.21.15.151:6814/31361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0604069460 con 0x7f06040669a0 2023-09-30T01:57:51.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f2a30066a70 msgr2=0x7f2a30068e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f2a30066a70 0x7f2a30068e80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2f0fdbf700 1 -- 172.21.15.151:0/2593063560 --> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f080aeee0 con 0x7f2eec066aa0 2023-09-30T01:57:51.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2efdffb700 1 -- 172.21.15.151:0/2593063560 <== osd.4 v2:172.21.15.172:6800/31451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f080aeee0 con 0x7f2eec066aa0 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a300606a0 msgr2=0x7f2a30062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a300606a0 0x7f2a30062b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009d10 tx=0x7f2a480091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 msgr2=0x7f2a4c1519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a52732700 1 --2- 172.21.15.151:0/334877108 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a4c0ac510 0x7f2a4c1519b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00b610 tx=0x7f2a3c009120 comp rx=0 tx=0).stop 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.692+0000 7f2a51f31700 1 -- 172.21.15.151:0/334877108 reap_dead start 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.696+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 shutdown_connections 2023-09-30T01:57:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.696+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 >> 172.21.15.151:0/334877108 conn(0x7f2a4c070510 msgr2=0x7f2a4c143250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.696+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 shutdown_connections 2023-09-30T01:57:51.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.696+0000 7f2a52732700 1 -- 172.21.15.151:0/334877108 wait complete. 2023-09-30T01:57:51.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f2eec066aa0 msgr2=0x7f2eec068eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f2eec066aa0 0x7f2eec068eb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2eec0606d0 msgr2=0x7f2eec062b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2eec0606d0 0x7f2eec062b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2f04006f00 tx=0x7f2f04008040 comp rx=0 tx=0).stop 2023-09-30T01:57:51.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 msgr2=0x7f2f080b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 --2- 172.21.15.151:0/2593063560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f081424a0 0x7f2f080b0580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009ae0 tx=0x7f2ef8017000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2f0e35c700 1 -- 172.21.15.151:0/2593063560 reap_dead start 2023-09-30T01:57:51.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 shutdown_connections 2023-09-30T01:57:51.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 >> 172.21.15.151:0/2593063560 conn(0x7f2f080708b0 msgr2=0x7f2f0813e320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 --> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3dd4000ee0 con 0x7f3de0066e70 2023-09-30T01:57:51.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 shutdown_connections 2023-09-30T01:57:51.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f2eeb7fe700 1 -- 172.21.15.151:0/2593063560 wait complete. 2023-09-30T01:57:51.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.700+0000 7f3de9ffb700 1 -- 172.21.15.151:0/2705159916 <== osd.5 v2:172.21.15.172:6810/31452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3dd4000ee0 con 0x7f3de0066e70 2023-09-30T01:57:51.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff988938700 1 -- 172.21.15.151:0/255300976 --> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff984098510 con 0x7ff9640669f0 2023-09-30T01:57:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f3de0066e70 msgr2=0x7f3de0069280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f3de0066e70 0x7f3de0069280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3de0060aa0 msgr2=0x7f3de0062f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3de0060aa0 0x7f3de0062f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3df4141d90 tx=0x7f3de4008040 comp rx=0 tx=0).stop 2023-09-30T01:57:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 msgr2=0x7f3df4142e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3ddf7fe700 1 --2- 172.21.15.151:0/2705159916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3df406b180 0x7f3df4142e60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009b20 tx=0x7f3df0017000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f3dfa409700 1 -- 172.21.15.151:0/2705159916 reap_dead start 2023-09-30T01:57:51.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff9727fc700 1 -- 172.21.15.151:0/255300976 <== osd.7 v2:172.21.15.172:6808/31450 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff984098510 con 0x7ff9640669f0 2023-09-30T01:57:51.786 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:51.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:57:51.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7ff9640669f0 msgr2=0x7ff964068e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7ff9640669f0 0x7ff964068e00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff964060620 msgr2=0x7ff964062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff964060620 0x7ff964062ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff974003be0 tx=0x7ff974007ec0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff984153d30 msgr2=0x7ff984154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 --2- 172.21.15.151:0/255300976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff984153d30 0x7ff984154180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff97c015000 tx=0x7ff97c016000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff982d9d700 1 -- 172.21.15.151:0/255300976 reap_dead start 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 shutdown_connections 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 >> 172.21.15.151:0/255300976 conn(0x7ff984070fe0 msgr2=0x7ff9840af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 shutdown_connections 2023-09-30T01:57:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7ff963fff700 1 -- 172.21.15.151:0/255300976 wait complete. 2023-09-30T01:57:51.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.704+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 --> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0618140b50 con 0x7f06040669a0 2023-09-30T01:57:51.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 shutdown_connections 2023-09-30T01:57:51.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 >> 172.21.15.151:0/2705159916 conn(0x7f3df40687a0 msgr2=0x7f3df406e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 shutdown_connections 2023-09-30T01:57:51.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f05fffff700 1 -- 172.21.15.151:0/4064607366 <== osd.0 v2:172.21.15.151:6814/31361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0618140b50 con 0x7f06040669a0 2023-09-30T01:57:51.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f3ddf7fe700 1 -- 172.21.15.151:0/2705159916 wait complete. 2023-09-30T01:57:51.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 --> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd29c0adb90 con 0x7fd29c0a4020 2023-09-30T01:57:51.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7f06040669a0 msgr2=0x7f0604068db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7f06040669a0 0x7f0604068db0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd282ffd700 1 -- 172.21.15.151:0/2141000685 <== osd.3 v2:172.21.15.151:6800/31360 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd29c0adb90 con 0x7fd29c0a4020 2023-09-30T01:57:51.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f06040605d0 msgr2=0x7f0604062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f06040605d0 0x7f0604062a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0618141bb0 tx=0x7f060c007d50 comp rx=0 tx=0).stop 2023-09-30T01:57:51.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 msgr2=0x7f06181423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 --2- 172.21.15.151:0/4064607366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f061806b280 0x7f06181423d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f061000cc80 tx=0x7f0610001a90 comp rx=0 tx=0).stop 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061c893700 1 -- 172.21.15.151:0/4064607366 reap_dead start 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 shutdown_connections 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 >> 172.21.15.151:0/4064607366 conn(0x7f0618070510 msgr2=0x7f061814dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 shutdown_connections 2023-09-30T01:57:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7f061e2f6700 1 -- 172.21.15.151:0/4064607366 wait complete. 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7fd29c0a4020 msgr2=0x7fd29c13e420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7fd29c0a4020 0x7fd29c13e420 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd2880605d0 msgr2=0x7fd288062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd2880605d0 0x7fd288062a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009b10 tx=0x7fd28c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 msgr2=0x7fd29c150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 --2- 172.21.15.151:0/2141000685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd29c0ab740 0x7fd29c150f80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd29000ec50 tx=0x7fd2900091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd29b7fe700 1 -- 172.21.15.151:0/2141000685 reap_dead start 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 shutdown_connections 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 >> 172.21.15.151:0/2141000685 conn(0x7fd29c0af880 msgr2=0x7fd29c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 shutdown_connections 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fd2a17a0700 1 -- 172.21.15.151:0/2141000685 wait complete. 2023-09-30T01:57:51.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 --> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc504005bd0 con 0x7fc5040015f0 2023-09-30T01:57:51.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.708+0000 7fc520ff9700 1 -- 172.21.15.151:0/2544735421 <== osd.2 v2:172.21.15.151:6808/31362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc504005bd0 con 0x7fc5040015f0 2023-09-30T01:57:51.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7fc5040015f0 msgr2=0x7fc504003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7fc5040015f0 0x7fc504003aa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc510060650 msgr2=0x7fc510062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc510060650 0x7fc510062b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc524145960 tx=0x7fc518009040 comp rx=0 tx=0).stop 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 msgr2=0x7fc5240a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 --2- 172.21.15.151:0/2544735421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5240b01c0 0x7fc5240a4300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00b610 tx=0x7fc51c019000 comp rx=0 tx=0).stop 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc5294d1700 1 -- 172.21.15.151:0/2544735421 reap_dead start 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 shutdown_connections 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 >> 172.21.15.151:0/2544735421 conn(0x7fc524070fe0 msgr2=0x7fc5240aa940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 shutdown_connections 2023-09-30T01:57:51.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.712+0000 7fc52af34700 1 -- 172.21.15.151:0/2544735421 wait complete. 2023-09-30T01:57:51.799 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-09-30T01:57:51.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T01:57:51.800 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:51.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:57:51.802 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:51.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:57:51.803 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:51.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T01:57:51.804 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:51.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:57:51.805 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:51.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:57:51.807 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:51.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:57:51.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.980+0000 7f83ca14a700 1 Processor -- start 2023-09-30T01:57:51.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.980+0000 7f0cd1d45700 1 Processor -- start 2023-09-30T01:57:51.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.980+0000 7f83ca14a700 1 -- start start 2023-09-30T01:57:51.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.980+0000 7f0cd1d45700 1 -- start start 2023-09-30T01:57:51.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 Processor -- start 2023-09-30T01:57:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83ca14a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c40ac1a0 0x7f83c406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83ca14a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f83c4140c30 0x7f83c413d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83ca14a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c4141220 con 0x7f83c40ac570 2023-09-30T01:57:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83ca14a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c413de20 con 0x7f83c4140c30 2023-09-30T01:57:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83ca14a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c413ac20 con 0x7f83c40ac1a0 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc06d3f0 con 0x7f0ccc0acfd0 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc06e5d0 con 0x7f0ccc13cf80 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc13a5a0 con 0x7f0ccc0acc00 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55522/0 (socket says 172.21.15.151:55522) 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 -- 172.21.15.151:0/1639946071 learned_addr learned my addr 172.21.15.151:0/1639946071 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccb7fe700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 -- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 msgr2=0x7f0ccc06e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc06e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 -- 172.21.15.151:0/1639946071 >> v1:172.21.15.151:6789/0 conn(0x7f0ccc0acfd0 legacy=0x7f0ccc136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 -- 172.21.15.151:0/1639946071 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc06e840 con 0x7f0ccc13cf80 2023-09-30T01:57:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 Processor -- start 2023-09-30T01:57:51.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 -- start start 2023-09-30T01:57:51.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 -- start start 2023-09-30T01:57:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e406d3f0 con 0x7fb3e40acfd0 2023-09-30T01:57:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e406e5d0 con 0x7fb3e413cf80 2023-09-30T01:57:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e9437700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e413a5a0 con 0x7fb3e40acc00 2023-09-30T01:57:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44588/0 (socket says 172.21.15.151:44588) 2023-09-30T01:57:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 -- 172.21.15.151:0/3821553277 learned_addr learned my addr 172.21.15.151:0/3821553277 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9148700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c40ac1a0 0x7f83c406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c8947700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f83c40ac570 0x7f83c413d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41320/0 (socket says 172.21.15.151:41320) 2023-09-30T01:57:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c8947700 1 -- 172.21.15.151:0/1464448360 learned_addr learned my addr 172.21.15.151:0/1464448360 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00099b0 tx=0x7f0cc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b0971b8061289d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 --2- 172.21.15.151:0/1464448360 >> v2:172.21.15.151:3300/0 conn(0x7f83c4140c30 0x7f83c413d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 -- 172.21.15.151:0/3821553277 >> v1:172.21.15.172:6789/0 conn(0x7fb3e413cf80 legacy=0x7fb3e413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 -- 172.21.15.151:0/3821553277 >> v1:172.21.15.151:6789/0 conn(0x7fb3e40acfd0 legacy=0x7fb3e4136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 -- 172.21.15.151:0/3821553277 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e406e840 con 0x7fb3e40acc00 2023-09-30T01:57:51.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cca7fc700 1 -- 172.21.15.151:0/1639946071 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0009ea0 con 0x7f0ccc13cf80 2023-09-30T01:57:51.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cca7fc700 1 -- 172.21.15.151:0/1639946071 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc000b860 con 0x7f0ccc13cf80 2023-09-30T01:57:51.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cca7fc700 1 -- 172.21.15.151:0/1639946071 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000bc00 con 0x7f0ccc13cf80 2023-09-30T01:57:51.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f178c06b880 0x7f178c1373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c06e560 con 0x7f178c06b4b0 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c06d4e0 con 0x7f178c06e9f0 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 -- 172.21.15.151:0/1464448360 >> v2:172.21.15.172:3300/0 conn(0x7f83c40ac1a0 msgr2=0x7f83c406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 --2- 172.21.15.151:0/1464448360 >> v2:172.21.15.172:3300/0 conn(0x7f83c40ac1a0 0x7f83c406dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 -- 172.21.15.151:0/1464448360 >> v1:172.21.15.172:6789/0 conn(0x7f83c40ac570 legacy=0x7f83c413d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83bbfff700 1 -- 172.21.15.151:0/1464448360 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699190225 0 0) 0x7f83c4141220 con 0x7f83c40ac570 2023-09-30T01:57:51.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f1790a5e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178c06d7c0 con 0x7f178c06b880 2023-09-30T01:57:51.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 -- 172.21.15.151:0/1464448360 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c40abdb0 con 0x7f83c4140c30 2023-09-30T01:57:51.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fdec0cc4700 1 Processor -- start 2023-09-30T01:57:51.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178ad9d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f178c06b880 0x7f178c1373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:51.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178a59c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f178c06b4b0 0x7f178c139b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43322/0 (socket says 172.21.15.151:43322) 2023-09-30T01:57:51.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178a59c700 1 -- 172.21.15.151:0/3406731247 learned_addr learned my addr 172.21.15.151:0/3406731247 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:51.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fdec0cc4700 1 -- start start 2023-09-30T01:57:51.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83c9949700 1 --2- 172.21.15.151:0/1464448360 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c4140c30 0x7f83c413d8e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83c4070c90 tx=0x7f83c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dfafb1d7933f9a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e2ffd700 1 --2- 172.21.15.151:0/3821553277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400d050 tx=0x7fb3d40082d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c2d4901b18e5bd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83bbfff700 1 -- 172.21.15.151:0/1464448360 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0011630 con 0x7f83c4140c30 2023-09-30T01:57:52.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83bbfff700 1 -- 172.21.15.151:0/1464448360 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0011790 con 0x7f83c4140c30 2023-09-30T01:57:52.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f83bbfff700 1 -- 172.21.15.151:0/1464448360 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0011b30 con 0x7f83c4140c30 2023-09-30T01:57:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 -- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 msgr2=0x7f0ccc13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f0cd1d45700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00099b0 tx=0x7f0cc0009250 comp rx=0 tx=0).stop 2023-09-30T01:57:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178ad9d700 1 -- 172.21.15.151:0/3406731247 >> v1:172.21.15.172:6789/0 conn(0x7f178c06e9f0 legacy=0x7f178c136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178ad9d700 1 -- 172.21.15.151:0/3406731247 >> v1:172.21.15.151:6789/0 conn(0x7f178c06b4b0 legacy=0x7f178c139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178ad9d700 1 -- 172.21.15.151:0/3406731247 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c06c5e0 con 0x7f178c06b880 2023-09-30T01:57:52.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e1ffb700 1 -- 172.21.15.151:0/3821553277 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4010070 con 0x7fb3e40acc00 2023-09-30T01:57:52.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e1ffb700 1 -- 172.21.15.151:0/3821553277 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d400dae0 con 0x7fb3e40acc00 2023-09-30T01:57:52.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7fb3e1ffb700 1 -- 172.21.15.151:0/3821553277 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d400de80 con 0x7fb3e40acc00 2023-09-30T01:57:52.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7f178ad9d700 1 --2- 172.21.15.151:0/3406731247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b880 0x7f178c1373c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f177c00bb90 tx=0x7f177c00b430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb1688e5fc9507c3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 -- 172.21.15.151:0/1639946071 shutdown_connections 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc13d3e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 --2- 172.21.15.151:0/1639946071 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc06e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 -- 172.21.15.151:0/1639946071 >> 172.21.15.151:0/1639946071 conn(0x7f0ccc070510 msgr2=0x7f0ccc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 -- 172.21.15.151:0/1639946071 shutdown_connections 2023-09-30T01:57:52.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f178959a700 1 -- 172.21.15.151:0/3406731247 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c00dba0 con 0x7f178c06b880 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f178959a700 1 -- 172.21.15.151:0/3406731247 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c00dd00 con 0x7f178c06b880 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f178959a700 1 -- 172.21.15.151:0/3406731247 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c019690 con 0x7f178c06b880 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdec0cc4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdebc13d0e0 0x7fdebc13d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdec0cc4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 0x7fdebc139c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdec0cc4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc06bf60 con 0x7fdebc0accd0 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdec0cc4700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdebc06cf80 con 0x7fdebc139780 2023-09-30T01:57:52.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdec0cc4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdebc06c1b0 con 0x7fdebc13d0e0 2023-09-30T01:57:52.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 Processor -- start 2023-09-30T01:57:52.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 -- 172.21.15.151:0/1464448360 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c4140c30 msgr2=0x7f83c413d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 --2- 172.21.15.151:0/1464448360 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c4140c30 0x7f83c413d8e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83c4070c90 tx=0x7f83c00091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.984+0000 7efef359e700 1 Processor -- start 2023-09-30T01:57:52.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef359e700 1 -- start start 2023-09-30T01:57:52.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdebc13d0e0 0x7fdebc13d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 -- start start 2023-09-30T01:57:52.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 0x7fdebc139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdebc13d0e0 0x7fdebc13d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50822/0 (socket says 172.21.15.151:50822) 2023-09-30T01:57:52.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 -- 172.21.15.151:0/4156428252 learned_addr learned my addr 172.21.15.151:0/4156428252 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- 172.21.15.151:0/3821553277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 msgr2=0x7fb3e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 --2- 172.21.15.151:0/3821553277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400d050 tx=0x7fb3d40082d0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efef40add00 0x7efef406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 -- 172.21.15.151:0/1464448360 shutdown_connections 2023-09-30T01:57:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 --2- 172.21.15.151:0/1464448360 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c4140c30 0x7f83c413d8e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 --2- 172.21.15.151:0/1464448360 >> v2:172.21.15.172:3300/0 conn(0x7f83c40ac1a0 0x7f83c406dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 -- 172.21.15.151:0/1464448360 >> 172.21.15.151:0/1464448360 conn(0x7f83c40687a0 msgr2=0x7f83c4070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- 172.21.15.151:0/3821553277 shutdown_connections 2023-09-30T01:57:52.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 --2- 172.21.15.151:0/3821553277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acc00 0x7fb3e406e090 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400d050 tx=0x7fb3d40082d0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- 172.21.15.151:0/3821553277 >> 172.21.15.151:0/3821553277 conn(0x7fb3e4070510 msgr2=0x7fb3e4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef359e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef406af00 con 0x7efef40b4510 2023-09-30T01:57:52.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef359e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef40b48e0 con 0x7efef4149280 2023-09-30T01:57:52.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef406d560 con 0x7efef40add00 2023-09-30T01:57:52.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efef40add00 0x7efef406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- 172.21.15.151:0/3406731247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b880 msgr2=0x7f178c1373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3406731247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b880 0x7f178c1373c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f177c00bb90 tx=0x7f177c00b430 comp rx=0 tx=0).stop 2023-09-30T01:57:52.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efef40add00 0x7efef406b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55546/0 (socket says 172.21.15.151:55546) 2023-09-30T01:57:52.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 -- 172.21.15.151:0/2712623368 learned_addr learned my addr 172.21.15.151:0/2712623368 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- 172.21.15.151:0/3821553277 shutdown_connections 2023-09-30T01:57:52.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- 172.21.15.151:0/3821553277 wait complete. 2023-09-30T01:57:52.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 -- 172.21.15.151:0/4156428252 >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 msgr2=0x7fdebc139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:52.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- 172.21.15.151:0/3406731247 shutdown_connections 2023-09-30T01:57:52.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3406731247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b880 0x7f178c1373c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- 172.21.15.151:0/3406731247 >> 172.21.15.151:0/3406731247 conn(0x7f178c0708b0 msgr2=0x7f178c070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 -- 172.21.15.151:0/1464448360 shutdown_connections 2023-09-30T01:57:52.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- 172.21.15.151:0/3406731247 shutdown_connections 2023-09-30T01:57:52.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- 172.21.15.151:0/3406731247 wait complete. 2023-09-30T01:57:52.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 --2- 172.21.15.151:0/4156428252 >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 0x7fdebc139c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdebaffd700 1 -- 172.21.15.151:0/4156428252 >> v1:172.21.15.172:6789/0 conn(0x7fdebc0accd0 legacy=0x7fdebc06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc0dc0acfd0 0x7fc0dc136a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc0dc13cf80 0x7fc0dc13d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0dc06d3f0 con 0x7fc0dc0acc00 2023-09-30T01:57:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0dc13a5a0 con 0x7fc0dc13cf80 2023-09-30T01:57:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e3a6a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0dc06e840 con 0x7fc0dc0acfd0 2023-09-30T01:57:52.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 -- 172.21.15.151:0/2712623368 >> v1:172.21.15.172:6789/0 conn(0x7efef4149280 legacy=0x7efef414b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:52.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 -- 172.21.15.151:0/2712623368 >> v1:172.21.15.151:6789/0 conn(0x7efef40b4510 legacy=0x7efef406d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 -- 172.21.15.151:0/2712623368 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef406e090 con 0x7efef40add00 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc0dc13cf80 0x7fc0dc13d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e1806700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc0dc0acc00 0x7fc0dc06e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43334/0 (socket says 172.21.15.151:43334) 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e1806700 1 -- 172.21.15.151:0/3840611258 learned_addr learned my addr 172.21.15.151:0/3840611258 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e1005700 1 --2- 172.21.15.151:0/3840611258 >> v2:172.21.15.172:3300/0 conn(0x7fc0dc0acfd0 0x7fc0dc136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 -- 172.21.15.151:0/3840611258 >> v2:172.21.15.172:3300/0 conn(0x7fc0dc0acfd0 msgr2=0x7fc0dc136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 --2- 172.21.15.151:0/3840611258 >> v2:172.21.15.172:3300/0 conn(0x7fc0dc0acfd0 0x7fc0dc136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7efef1d9b700 1 --2- 172.21.15.151:0/2712623368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef40add00 0x7efef406b390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efee80099b0 tx=0x7efee8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235e3b6371a17b31 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 -- 172.21.15.151:0/3840611258 >> v1:172.21.15.151:6789/0 conn(0x7fc0dc0acc00 legacy=0x7fc0dc06e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 -- 172.21.15.151:0/3840611258 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc136fc0 con 0x7fc0dc13cf80 2023-09-30T01:57:52.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 -- 172.21.15.151:0/1639946071 wait complete. 2023-09-30T01:57:52.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 Processor -- start 2023-09-30T01:57:52.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f83ca14a700 1 -- 172.21.15.151:0/1464448360 wait complete. 2023-09-30T01:57:52.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 -- start start 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 Processor -- start 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- start start 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b4b0 0x7f178c13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f178c06b880 0x7f178c136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c134b60 con 0x7f178c06e9f0 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c136a40 con 0x7f178c06b4b0 2023-09-30T01:57:52.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f1790a5e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c135550 con 0x7f178c06b880 2023-09-30T01:57:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7f0cd1d45700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 Processor -- start 2023-09-30T01:57:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fb3e9437700 1 -- start start 2023-09-30T01:57:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acfd0 0x7fb3e4155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fdeba7fc700 1 -- 172.21.15.151:0/4156428252 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663595277 0 0) 0x7fdebc06bf60 con 0x7fdebc0accd0 2023-09-30T01:57:52.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.988+0000 7fc0e2007700 1 --2- 172.21.15.151:0/3840611258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc13cf80 0x7fc0dc13d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400d0d0 tx=0x7fc0d400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138a15b8cf08482c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3e40af3d0 0x7fb3e40b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4154f20 con 0x7fb3e40af3d0 2023-09-30T01:57:52.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40ae800 con 0x7fb3e413cf80 2023-09-30T01:57:52.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40a8880 con 0x7fb3e40acfd0 2023-09-30T01:57:52.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178a59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b4b0 0x7f178c13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 Processor -- start 2023-09-30T01:57:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- start start 2023-09-30T01:57:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e0804700 1 -- 172.21.15.151:0/3840611258 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4010070 con 0x7fc0dc13cf80 2023-09-30T01:57:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3e40af3d0 0x7fb3e40b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50852/0 (socket says 172.21.15.151:50852) 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 -- 172.21.15.151:0/1758768135 learned_addr learned my addr 172.21.15.151:0/1758768135 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ccc0acfd0 0x7f0ccc1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0a5410 con 0x7f0ccc13cf80 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc151f60 con 0x7f0ccc0acfd0 2023-09-30T01:57:52.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0a8ed0 con 0x7f0ccc0acc00 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc0a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44592/0 (socket says 172.21.15.151:44592) 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccb7fe700 1 -- 172.21.15.151:0/3874877867 learned_addr learned my addr 172.21.15.151:0/3874877867 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55572/0 (socket says 172.21.15.151:55572) 2023-09-30T01:57:52.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 -- 172.21.15.151:0/3218865466 learned_addr learned my addr 172.21.15.151:0/3218865466 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdebaffd700 1 -- 172.21.15.151:0/4156428252 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc0ad0a0 con 0x7fdebc13d0e0 2023-09-30T01:57:52.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdebbfff700 1 --2- 172.21.15.151:0/4156428252 >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 0x7fdebc139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:52.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e0804700 1 -- 172.21.15.151:0/3840611258 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d400ddb0 con 0x7fc0dc13cf80 2023-09-30T01:57:52.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e0804700 1 -- 172.21.15.151:0/3840611258 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400b300 con 0x7fc0dc13cf80 2023-09-30T01:57:52.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccaffd700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ccc0acfd0 0x7f0ccc1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 -- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f178c06b880 msgr2=0x7f178c136500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f178c06b880 0x7f178c136500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 -- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b4b0 msgr2=0x7f178c13cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b4b0 0x7f178c13cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 -- 172.21.15.151:0/3218865466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c00b6d0 con 0x7f178c06e9f0 2023-09-30T01:57:52.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef159a700 1 -- 172.21.15.151:0/2712623368 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800b820 con 0x7efef40add00 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef159a700 1 -- 172.21.15.151:0/2712623368 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee800b980 con 0x7efef40add00 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef159a700 1 -- 172.21.15.151:0/2712623368 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800bd00 con 0x7efef40add00 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 -- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 msgr2=0x7f0ccc0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 -- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ccc0acfd0 msgr2=0x7f0ccc1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ccc0acfd0 0x7f0ccc1514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 -- 172.21.15.151:0/3874877867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc00094f0 con 0x7f0ccc13cf80 2023-09-30T01:57:52.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f178ad9d700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f177c00b470 tx=0x7f177c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdebaffd700 1 --2- 172.21.15.151:0/4156428252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc13d510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdebc06c7f0 tx=0x7fdeb000b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c99e91fbec8d8625 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 -- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acfd0 msgr2=0x7fb3e4155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acfd0 0x7fb3e4155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 -- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3e40af3d0 msgr2=0x7fb3e40b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3e40af3d0 0x7fb3e40b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 -- 172.21.15.151:0/1758768135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d4008570 con 0x7fb3e413cf80 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c40ac1a0 0x7f83c40a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c00de70 con 0x7f178c06e9f0 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 -- 172.21.15.151:0/2712623368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef40add00 msgr2=0x7efef406b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 --2- 172.21.15.151:0/2712623368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef40add00 0x7efef406b390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efee80099b0 tx=0x7efee8009250 comp rx=0 tx=0).stop 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 -- 172.21.15.151:0/2712623368 shutdown_connections 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 --2- 172.21.15.151:0/2712623368 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef40add00 0x7efef406b390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 -- 172.21.15.151:0/2712623368 >> 172.21.15.151:0/2712623368 conn(0x7efef4070f90 msgr2=0x7efef40b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c40ac570 0x7f83c4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40a4bb0 con 0x7f83c40ac570 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40a4e50 con 0x7f83c40ac1a0 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40a5ca0 con 0x7f83c4140c30 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9148700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c40ac1a0 0x7f83c40a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44600/0 (socket says 172.21.15.151:44600) 2023-09-30T01:57:52.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 -- 172.21.15.151:0/2828909208 learned_addr learned my addr 172.21.15.151:0/2828909208 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c8947700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c40ac570 0x7f83c4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/3840611258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc13cf80 msgr2=0x7fc0dc13d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/3840611258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc13cf80 0x7fc0dc13d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400d0d0 tx=0x7fc0d400b990 comp rx=0 tx=0).stop 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/3840611258 shutdown_connections 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/3840611258 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc13cf80 0x7fc0dc13d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/3840611258 >> v2:172.21.15.172:3300/0 conn(0x7fc0dc0acfd0 0x7fc0dc136a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/3840611258 >> 172.21.15.151:0/3840611258 conn(0x7fc0dc070510 msgr2=0x7fc0dc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e27fc700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800c5d0 tx=0x7fb3d8007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8011590 con 0x7fb3e413cf80 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d80116f0 con 0x7fb3e413cf80 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8011ab0 con 0x7fb3e413cf80 2023-09-30T01:57:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e40a8b60 con 0x7fb3e413cf80 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e4155ad0 con 0x7fb3e413cf80 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0ccbfff700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc151a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0b0cd0 tx=0x7f0cc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0011420 con 0x7f0ccc13cf80 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c010b00 con 0x7f178c06e9f0 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c024420 con 0x7f178c06e9f0 2023-09-30T01:57:52.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c135830 con 0x7f178c06e9f0 2023-09-30T01:57:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c133980 con 0x7f178c06e9f0 2023-09-30T01:57:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdeba7fc700 1 -- 172.21.15.151:0/4156428252 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000dba0 con 0x7fdebc13d0e0 2023-09-30T01:57:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdeba7fc700 1 -- 172.21.15.151:0/4156428252 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb000dd00 con 0x7fdebc13d0e0 2023-09-30T01:57:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdeba7fc700 1 -- 172.21.15.151:0/4156428252 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00196d0 con 0x7fdebc13d0e0 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 -- 172.21.15.151:0/2712623368 shutdown_connections 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7efef359e700 1 -- 172.21.15.151:0/2712623368 wait complete. 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 Processor -- start 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 -- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c40ac1a0 msgr2=0x7f83c40a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c40ac1a0 0x7f83c40a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 -- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c40ac570 msgr2=0x7f83c4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c40ac570 0x7f83c4141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 -- 172.21.15.151:0/2828909208 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c0009420 con 0x7f83c4140c30 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83c9949700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f83c4070cc0 tx=0x7f83c0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0011a50 con 0x7f83c4140c30 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0026430 con 0x7f83c4140c30 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c40a5fe0 con 0x7f83c4140c30 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c4142600 con 0x7f83c4140c30 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f177c024810 con 0x7f178c06e9f0 2023-09-30T01:57:52.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f177a7fc700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 0x7f176c062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f177c020070 con 0x7f178c06e9f0 2023-09-30T01:57:52.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdec0cc4700 1 -- 172.21.15.151:0/4156428252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 msgr2=0x7fdebc13d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/4156428252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc13d510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdebc06c7f0 tx=0x7fdeb000b4f0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c00267f0 con 0x7f83c4140c30 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 -- 172.21.15.151:0/3874877867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc129400 con 0x7f0ccc13cf80 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cd1d45700 1 -- 172.21.15.151:0/3874877867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc0a7950 con 0x7f0ccc13cf80 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc0011790 con 0x7f0ccc13cf80 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0011900 con 0x7f0ccc13cf80 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0cd1d45700 1 -- 172.21.15.151:0/3874877867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac0050b0 con 0x7f0ccc13cf80 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/3840611258 shutdown_connections 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.992+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/3840611258 wait complete. 2023-09-30T01:57:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 Processor -- start 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 -- start start 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fdec0cc4700 1 -- 172.21.15.151:0/4156428252 shutdown_connections 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/4156428252 >> v2:172.21.15.151:3300/0 conn(0x7fdebc139780 0x7fdebc139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/4156428252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc13d510 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fdec0cc4700 1 -- 172.21.15.151:0/4156428252 >> 172.21.15.151:0/4156428252 conn(0x7fdebc070510 msgr2=0x7fdebc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a40050b0 con 0x7f83c4140c30 2023-09-30T01:57:52.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e40670a0 con 0x7fb3e413cf80 2023-09-30T01:57:52.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 -- start start 2023-09-30T01:57:52.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 0x7efef414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 0x7efef414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 0x7efef414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 0x7efef414fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50880/0 (socket says 172.21.15.151:50880) 2023-09-30T01:57:52.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0cc0011a60 con 0x7f0ccc13cf80 2023-09-30T01:57:52.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f83c0023030 con 0x7f83c4140c30 2023-09-30T01:57:52.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44628/0 (socket says 172.21.15.151:44628) 2023-09-30T01:57:52.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef259c700 1 -- 172.21.15.151:0/104398771 learned_addr learned my addr 172.21.15.151:0/104398771 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef4150590 con 0x7efef4149280 2023-09-30T01:57:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef2d9d700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 0x7efef414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef41502b0 con 0x7efef40add00 2023-09-30T01:57:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef414b0d0 con 0x7efef40b4510 2023-09-30T01:57:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0cb37fe700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 0x7f0cb8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83b97fa700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 0x7f83a8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0cc007dd50 con 0x7f0ccc13cf80 2023-09-30T01:57:52.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f83c007df60 con 0x7f83c4140c30 2023-09-30T01:57:52.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0acfd0 0x7fc0dc0a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0dc13cf80 0x7fc0dc0a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0aa310 con 0x7fc0dc0acc00 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0b0630 con 0x7fc0dc0acfd0 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e3a6a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc086760 con 0x7fc0dc13cf80 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55610/0 (socket says 172.21.15.151:55610) 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 -- 172.21.15.151:0/2109085157 learned_addr learned my addr 172.21.15.151:0/2109085157 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 -- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 msgr2=0x7efef414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 0x7efef414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 -- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 msgr2=0x7efef414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 0x7efef414ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e2007700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0dc13cf80 0x7fc0dc0a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1005700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0acfd0 0x7fc0dc0a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80094f0 con 0x7efef40b4510 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef2d9d700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 0x7efef414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 -- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0dc13cf80 msgr2=0x7fc0dc0a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0dc13cf80 0x7fc0dc0a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 -- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0acfd0 msgr2=0x7fc0dc0a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0acfd0 0x7fc0dc0a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 -- 172.21.15.151:0/2109085157 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d400bc30 con 0x7fc0dc0acc00 2023-09-30T01:57:52.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7fc0e1806700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0d801d570 tx=0x7fc0d8001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f83c9148700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 0x7f83a8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f0ccb7fe700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 0x7f0cb8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f178a59c700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 0x7f176c062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7efef1d9b700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efef406ba30 tx=0x7efee8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:51.996+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c136e80 con 0x7f178c06e9f0 2023-09-30T01:57:52.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7f0ccb7fe700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 0x7f0cb8062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009e70 tx=0x7f0cbc0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fdec0cc4700 1 -- 172.21.15.151:0/4156428252 shutdown_connections 2023-09-30T01:57:52.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb3d8036b50 con 0x7fb3e413cf80 2023-09-30T01:57:52.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7f83c9148700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 0x7f83a8062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009db0 tx=0x7f83b40091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7f178a59c700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 0x7f176c062a80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1784002040 tx=0x7f1784009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb3caffd700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 0x7fb3d0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb3d807c2e0 con 0x7fb3e413cf80 2023-09-30T01:57:52.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb3e2ffd700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 0x7fb3d0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb3e2ffd700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 0x7fb3d0062a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400d050 tx=0x7fb3d401b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fdec0cc4700 1 -- 172.21.15.151:0/4156428252 wait complete. 2023-09-30T01:57:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb2d4b19700 1 Processor -- start 2023-09-30T01:57:52.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb2d4b19700 1 -- start start 2023-09-30T01:57:52.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fdec0cc4700 1 Processor -- start 2023-09-30T01:57:52.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8005b30 con 0x7fc0dc0acc00 2023-09-30T01:57:52.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0cc004e490 con 0x7f0ccc13cf80 2023-09-30T01:57:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fdec0cc4700 1 -- start start 2023-09-30T01:57:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8024440 con 0x7efef40b4510 2023-09-30T01:57:52.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f83c00a7be0 con 0x7f83c4140c30 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.000+0000 7fb2d4b19700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb2d006ec30 0x7fb2d01375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2d4b19700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d006e0c0 con 0x7fb2d006bc60 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2d4b19700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00abcc0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2d4b19700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d006e750 con 0x7fb2d006ec30 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2ced9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb2d006ec30 0x7fb2d01375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2cdd9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb2d006bc60 0x7fb2d0136ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43350/0 (socket says 172.21.15.151:43350) 2023-09-30T01:57:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2ce59c700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb2d00ab8f0 0x7fb2d006c7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41376/0 (socket says 172.21.15.151:41376) 2023-09-30T01:57:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2cdd9b700 1 -- 172.21.15.151:0/3095739633 learned_addr learned my addr 172.21.15.151:0/3095739633 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef414f640 con 0x7efef40b4510 2023-09-30T01:57:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef414c3a0 con 0x7efef40b4510 2023-09-30T01:57:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee80245a0 con 0x7efef40b4510 2023-09-30T01:57:52.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8024940 con 0x7efef40b4510 2023-09-30T01:57:52.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc0868a0 con 0x7fc0dc0acc00 2023-09-30T01:57:52.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc0a6640 con 0x7fc0dc0acc00 2023-09-30T01:57:52.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efee801a430 con 0x7efef40b4510 2023-09-30T01:57:52.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 0x7fdebc155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc0a7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc155850 con 0x7fdebc139780 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc1555c0 con 0x7fdebc13d0e0 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc14eeb0 con 0x7fdebc0accd0 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44638/0 (socket says 172.21.15.151:44638) 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3455462504 learned_addr learned my addr 172.21.15.151:0/3455462504 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebaffd700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 0x7fdebc155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 msgr2=0x7fdebc0a7820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc0a7820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 msgr2=0x7fdebc155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 0x7fdebc155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 -- 172.21.15.151:0/3455462504 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4009a20 con 0x7fdebc0accd0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebaffd700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 0x7fdebc155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdebb7fe700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00c7e0 tx=0x7fdeac007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac011590 con 0x7fdebc0accd0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac0116f0 con 0x7fdebc0accd0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb000b790 con 0x7fdebc0accd0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac011ab0 con 0x7fdebc0accd0 2023-09-30T01:57:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc0a7d60 con 0x7fdebc0accd0 2023-09-30T01:57:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc0670a0 con 0x7fc0dc0acc00 2023-09-30T01:57:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed80050b0 con 0x7efef40b4510 2023-09-30T01:57:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efee27fc700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 0x7efed4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efee807e5e0 con 0x7efef40b4510 2023-09-30T01:57:52.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7efef259c700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 0x7efed4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2ced9d700 1 -- 172.21.15.151:0/3095739633 >> v1:172.21.15.172:6789/0 conn(0x7fb2d00ab8f0 legacy=0x7fb2d006c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2ced9d700 1 -- 172.21.15.151:0/3095739633 >> v1:172.21.15.151:6789/0 conn(0x7fb2d006bc60 legacy=0x7fb2d0136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.004+0000 7fb2ced9d700 1 -- 172.21.15.151:0/3095739633 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d006e330 con 0x7fb2d006ec30 2023-09-30T01:57:52.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdeac01b3b0 con 0x7fdebc0accd0 2023-09-30T01:57:52.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7efef259c700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 0x7efed4062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efee4007fd0 tx=0x7efee4000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fdea37fe700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 0x7fdea8062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fb2ced9d700 1 --2- 172.21.15.151:0/3095739633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006ec30 0x7fb2d01375e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400de60 tx=0x7fb2c400d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b54863aa72168ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdebc0a74e0 con 0x7fdebc0accd0 2023-09-30T01:57:52.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fdebaffd700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 0x7fdea8062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb3d804cb40 con 0x7fb3e413cf80 2023-09-30T01:57:52.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.008+0000 7fdebaffd700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 0x7fdea8062a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdebc06c7f0 tx=0x7fdeb001b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f177c04f230 con 0x7f178c06e9f0 2023-09-30T01:57:52.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7fb2cd59a700 1 -- 172.21.15.151:0/3095739633 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4012070 con 0x7fb2d006ec30 2023-09-30T01:57:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efee804edc0 con 0x7efef40b4510 2023-09-30T01:57:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7fb2cd59a700 1 -- 172.21.15.151:0/3095739633 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c400fc30 con 0x7fb2d006ec30 2023-09-30T01:57:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7fb2cd59a700 1 -- 172.21.15.151:0/3095739633 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400aea0 con 0x7fb2d006ec30 2023-09-30T01:57:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.012+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fdeac050460 con 0x7fdebc0accd0 2023-09-30T01:57:52.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.016+0000 7fb2d4b19700 1 -- 172.21.15.151:0/3095739633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006ec30 msgr2=0x7fb2d01375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.016+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/3095739633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006ec30 0x7fb2d01375e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400de60 tx=0x7fb2c400d780 comp rx=0 tx=0).stop 2023-09-30T01:57:52.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d8007070 con 0x7fc0dc0acc00 2023-09-30T01:57:52.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8005ea0 con 0x7fc0dc0acc00 2023-09-30T01:57:52.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc0d8007200 con 0x7fc0dc0acc00 2023-09-30T01:57:52.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fb2d4b19700 1 -- 172.21.15.151:0/3095739633 shutdown_connections 2023-09-30T01:57:52.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/3095739633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006ec30 0x7fb2d01375e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fb2d4b19700 1 -- 172.21.15.151:0/3095739633 >> 172.21.15.151:0/3095739633 conn(0x7fb2d0070510 msgr2=0x7fb2d0070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fb2d4b19700 1 -- 172.21.15.151:0/3095739633 shutdown_connections 2023-09-30T01:57:52.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fb2d4b19700 1 -- 172.21.15.151:0/3095739633 wait complete. 2023-09-30T01:57:52.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 0x7fc0c0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc0d808d090 con 0x7fc0dc0acc00 2023-09-30T01:57:52.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.020+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc0d805d920 con 0x7fc0dc0acc00 2023-09-30T01:57:52.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 Processor -- start 2023-09-30T01:57:52.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fc0e1005700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 0x7fc0c0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 -- start start 2023-09-30T01:57:52.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 0x7fb2d0148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ce59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ce59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55658/0 (socket says 172.21.15.151:55658) 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44650/0 (socket says 172.21.15.151:44650) 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2cdd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 0x7fb2d0148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdeac081060 con 0x7fdebc0accd0 2023-09-30T01:57:52.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2cdd9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 0x7fb2d0148860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50900/0 (socket says 172.21.15.151:50900) 2023-09-30T01:57:52.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ce59c700 1 -- 172.21.15.151:0/1961526059 learned_addr learned my addr 172.21.15.151:0/1961526059 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:52.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d013de10 con 0x7fb2d006bc60 2023-09-30T01:57:52.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d013e2a0 con 0x7fb2d006ec30 2023-09-30T01:57:52.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d01492e0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 -- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 msgr2=0x7fb2d0148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 0x7fb2d0148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 -- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 msgr2=0x7fb2d013a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c400da20 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ce59c700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.024+0000 7fb2ced9d700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2c401a000 tx=0x7fb2c401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.028+0000 7fc0e1005700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 0x7fc0c0062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0a7d80 tx=0x7fc0cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4016d40 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c402a3f0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d013b090 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c402a7b0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d013dac0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb2c40323f0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.036+0000 7fb2be7fc700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 0x7fb2b0062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.040+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb2c4081da0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.040+0000 7fb2ce59c700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 0x7fb2b0062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.040+0000 7fb2ce59c700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 0x7fb2b0062ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0007060 tx=0x7fb2c0007f70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.044+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b40050b0 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.056+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb2c4052600 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.300+0000 7f0cd1d45700 1 -- 172.21.15.151:0/3874877867 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0cac005360 con 0x7f0ccc13cf80 2023-09-30T01:57:52.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.300+0000 7f0cb37fe700 1 -- 172.21.15.151:0/3874877867 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cc0015070 con 0x7f0ccc13cf80 2023-09-30T01:57:52.307 INFO:teuthology.orchestra.run.smithi151.stdout:60129542147 2023-09-30T01:57:52.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 msgr2=0x7f0cb8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 0x7f0cb8062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009e70 tx=0x7f0cbc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 msgr2=0x7f0ccc151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc151a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0b0cd0 tx=0x7f0cc0017000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 shutdown_connections 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cb8060550 0x7f0cb8062a00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ccc13cf80 0x7f0ccc151a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ccc0acfd0 0x7f0ccc1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 --2- 172.21.15.151:0/3874877867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ccc0acc00 0x7f0ccc0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 >> 172.21.15.151:0/3874877867 conn(0x7f0ccc070510 msgr2=0x7f0ccc06be20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 shutdown_connections 2023-09-30T01:57:52.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7f0cb17fa700 1 -- 172.21.15.151:0/3874877867 wait complete. 2023-09-30T01:57:52.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efed8005630 con 0x7efef40b4510 2023-09-30T01:57:52.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.304+0000 7efee27fc700 1 -- 172.21.15.151:0/104398771 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efee80523e0 con 0x7efef40b4510 2023-09-30T01:57:52.314 INFO:teuthology.orchestra.run.smithi151.stdout:55834574850 2023-09-30T01:57:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f83a4005630 con 0x7f83c4140c30 2023-09-30T01:57:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 msgr2=0x7efed4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 0x7efed4062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efee4007fd0 tx=0x7efee4000bc0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 msgr2=0x7efef414a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efef406ba30 tx=0x7efee8016000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7f83b97fa700 1 -- 172.21.15.151:0/2828909208 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f83c004e7a0 con 0x7f83c4140c30 2023-09-30T01:57:52.317 INFO:teuthology.orchestra.run.smithi151.stdout:55834574850 2023-09-30T01:57:52.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 shutdown_connections 2023-09-30T01:57:52.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efed4060550 0x7efed4062a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efef4149280 0x7efef414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efef40b4510 0x7efef414a620 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 --2- 172.21.15.151:0/104398771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efef40add00 0x7efef414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 msgr2=0x7f83a8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 >> 172.21.15.151:0/104398771 conn(0x7efef4070f90 msgr2=0x7efef41468d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 shutdown_connections 2023-09-30T01:57:52.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.312+0000 7efef359e700 1 -- 172.21.15.151:0/104398771 wait complete. 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 0x7f83a8062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009db0 tx=0x7f83b40091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 msgr2=0x7f83c41420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f83c4070cc0 tx=0x7f83c0017000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 shutdown_connections 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f83a80606a0 0x7f83a8062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f83c4140c30 0x7f83c41420c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f83c40ac570 0x7f83c4141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 --2- 172.21.15.151:0/2828909208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c40ac1a0 0x7f83c40a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 >> 172.21.15.151:0/2828909208 conn(0x7f83c40687a0 msgr2=0x7f83c406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 shutdown_connections 2023-09-30T01:57:52.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f83ca14a700 1 -- 172.21.15.151:0/2828909208 wait complete. 2023-09-30T01:57:52.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.316+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f178c0670a0 con 0x7f178c06e9f0 2023-09-30T01:57:52.327 INFO:teuthology.orchestra.run.smithi151.stdout:55834574850 2023-09-30T01:57:52.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.320+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc0dc06ee70 con 0x7fc0dc0acc00 2023-09-30T01:57:52.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.320+0000 7fc0d1ffb700 1 -- 172.21.15.151:0/2109085157 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0d8060f40 con 0x7fc0dc0acc00 2023-09-30T01:57:52.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.6 2023-09-30T01:57:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 msgr2=0x7fc0c0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 0x7fc0c0062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0a7d80 tx=0x7fc0cc008040 comp rx=0 tx=0).stop 2023-09-30T01:57:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 msgr2=0x7fc0dc0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0d801d570 tx=0x7fc0d8001020 comp rx=0 tx=0).stop 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7f177a7fc700 1 -- 172.21.15.151:0/3218865466 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f177c052850 con 0x7f178c06e9f0 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 shutdown_connections 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc0c0060580 0x7fc0c0062a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0dc13cf80 0x7fc0dc0a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0dc0acfd0 0x7fc0dc0a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 --2- 172.21.15.151:0/2109085157 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0dc0acc00 0x7fc0dc0afb90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 >> 172.21.15.151:0/2109085157 conn(0x7fc0dc070510 msgr2=0x7fc0dc06b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.331 INFO:teuthology.orchestra.run.smithi151.stdout:55834574849 2023-09-30T01:57:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 shutdown_connections 2023-09-30T01:57:52.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fc0e3a6a700 1 -- 172.21.15.151:0/2109085157 wait complete. 2023-09-30T01:57:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb3e413daf0 con 0x7fb3e413cf80 2023-09-30T01:57:52.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.324+0000 7fb3caffd700 1 -- 172.21.15.151:0/1758768135 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3d8050160 con 0x7fb3e413cf80 2023-09-30T01:57:52.332 INFO:teuthology.orchestra.run.smithi151.stdout:55834574849 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 msgr2=0x7f176c062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 0x7f176c062a80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1784002040 tx=0x7f1784009040 comp rx=0 tx=0).stop 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 msgr2=0x7f178c133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f177c00b470 tx=0x7f177c01b000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 shutdown_connections 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f176c0605d0 0x7f176c062a80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f178c06e9f0 0x7f178c133440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f178c06b880 0x7f178c136500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 --2- 172.21.15.151:0/3218865466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f178c06b4b0 0x7f178c13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 >> 172.21.15.151:0/3218865466 conn(0x7f178c0708b0 msgr2=0x7f178c070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 shutdown_connections 2023-09-30T01:57:52.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7f1790a5e700 1 -- 172.21.15.151:0/3218865466 wait complete. 2023-09-30T01:57:52.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.1 2023-09-30T01:57:52.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 msgr2=0x7fb3d0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 0x7fb3d0062a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400d050 tx=0x7fb3d401b000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 msgr2=0x7fb3e40b00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800c5d0 tx=0x7fb3d8007f60 comp rx=0 tx=0).stop 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 shutdown_connections 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb3d0060580 0x7fb3d0062a30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3e40af3d0 0x7fb3e40b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3e413cf80 0x7fb3e40b00c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 --2- 172.21.15.151:0/1758768135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3e40acfd0 0x7fb3e4155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 >> 172.21.15.151:0/1758768135 conn(0x7fb3e4070510 msgr2=0x7fb3e406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 shutdown_connections 2023-09-30T01:57:52.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.328+0000 7fb3e9437700 1 -- 172.21.15.151:0/1758768135 wait complete. 2023-09-30T01:57:52.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.332+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdebc0008d0 con 0x7fdebc0accd0 2023-09-30T01:57:52.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.332+0000 7fdea37fe700 1 -- 172.21.15.151:0/3455462504 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdeac017020 con 0x7fdebc0accd0 2023-09-30T01:57:52.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.3 2023-09-30T01:57:52.345 INFO:teuthology.orchestra.run.smithi151.stdout:55834574850 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 msgr2=0x7fdea8062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 0x7fdea8062a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdebc06c7f0 tx=0x7fdeb001b000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 msgr2=0x7fdebc154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00c7e0 tx=0x7fdeac007df0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 shutdown_connections 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdea80605d0 0x7fdea8062a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdebc13d0e0 0x7fdebc0a7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdebc139780 0x7fdebc155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 --2- 172.21.15.151:0/3455462504 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdebc0accd0 0x7fdebc154b40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 >> 172.21.15.151:0/3455462504 conn(0x7fdebc070510 msgr2=0x7fdebc0af780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 shutdown_connections 2023-09-30T01:57:52.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.336+0000 7fdec0cc4700 1 -- 172.21.15.151:0/3455462504 wait complete. 2023-09-30T01:57:52.348 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574849 for osd.7 2023-09-30T01:57:52.349 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.4 2023-09-30T01:57:52.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574849 for osd.5 2023-09-30T01:57:52.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb2b4005630 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2be7fc700 1 -- 172.21.15.151:0/1961526059 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb2c4055c20 con 0x7fb2d00ab8f0 2023-09-30T01:57:52.353 INFO:teuthology.orchestra.run.smithi151.stdout:55834574849 2023-09-30T01:57:52.356 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.0 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 msgr2=0x7fb2b0062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 0x7fb2b0062ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0007060 tx=0x7fb2c0007f70 comp rx=0 tx=0).stop 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 msgr2=0x7fb2d0148da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2c401a000 tx=0x7fb2c401d000 comp rx=0 tx=0).stop 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 shutdown_connections 2023-09-30T01:57:52.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb2b0060620 0x7fb2b0062ad0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2d00ab8f0 0x7fb2d0148da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d006ec30 0x7fb2d0148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 --2- 172.21.15.151:0/1961526059 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2d006bc60 0x7fb2d013a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 >> 172.21.15.151:0/1961526059 conn(0x7fb2d0070510 msgr2=0x7fb2d00b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 shutdown_connections 2023-09-30T01:57:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:52.348+0000 7fb2d4b19700 1 -- 172.21.15.151:0/1961526059 wait complete. 2023-09-30T01:57:52.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574849 for osd.2 2023-09-30T01:57:53.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T01:57:53.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:57:53.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:57:53.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T01:57:53.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:57:53.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:57:53.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:57:53.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:57:53.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.476+0000 7f2c06531700 1 Processor -- start 2023-09-30T01:57:53.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.476+0000 7f2c06531700 1 -- start start 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2c000acc00 0x7f2c0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 0x7f2c0013d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0006d3f0 con 0x7f2c000acfd0 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0006e5d0 con 0x7f2c000acc00 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0013a5a0 con 0x7f2c0013cf80 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2c000acc00 0x7f2c0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2c000acc00 0x7f2c0006e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55666/0 (socket says 172.21.15.151:55666) 2023-09-30T01:57:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 -- 172.21.15.151:0/1150191394 learned_addr learned my addr 172.21.15.151:0/1150191394 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1150191394 >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 0x7f2c0013d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 -- 172.21.15.151:0/1150191394 >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 msgr2=0x7f2c0013d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 --2- 172.21.15.151:0/1150191394 >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 0x7f2c0013d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 -- 172.21.15.151:0/1150191394 >> v1:172.21.15.172:6789/0 conn(0x7f2c000acfd0 legacy=0x7f2c00136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 -- 172.21.15.151:0/1150191394 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0006e840 con 0x7f2c000acc00 2023-09-30T01:57:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1150191394 >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 0x7f2c0013d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:53.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfffff700 1 --2- 172.21.15.151:0/1150191394 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c000acc00 0x7f2c0006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80089b0 tx=0x7f2bf80082d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0b20fc2119816e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfeffd700 1 -- 172.21.15.151:0/1150191394 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8010070 con 0x7f2c000acc00 2023-09-30T01:57:53.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfeffd700 1 -- 172.21.15.151:0/1150191394 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf800dab0 con 0x7f2c000acc00 2023-09-30T01:57:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2bfeffd700 1 -- 172.21.15.151:0/1150191394 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf800de50 con 0x7f2c000acc00 2023-09-30T01:57:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- 172.21.15.151:0/1150191394 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c000acc00 msgr2=0x7f2c0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 --2- 172.21.15.151:0/1150191394 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c000acc00 0x7f2c0006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80089b0 tx=0x7f2bf80082d0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- 172.21.15.151:0/1150191394 shutdown_connections 2023-09-30T01:57:53.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 --2- 172.21.15.151:0/1150191394 >> v2:172.21.15.172:3300/0 conn(0x7f2c0013cf80 0x7f2c0013d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 --2- 172.21.15.151:0/1150191394 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c000acc00 0x7f2c0006e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- 172.21.15.151:0/1150191394 >> 172.21.15.151:0/1150191394 conn(0x7f2c00070510 msgr2=0x7f2c00070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- 172.21.15.151:0/1150191394 shutdown_connections 2023-09-30T01:57:53.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7f2c06531700 1 -- 172.21.15.151:0/1150191394 wait complete. 2023-09-30T01:57:53.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7ffbf0b92700 1 Processor -- start 2023-09-30T01:57:53.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.480+0000 7ffbf0b92700 1 -- start start 2023-09-30T01:57:53.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffbec0a2f70 0x7ffbec0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec0a8ca0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec0a6660 con 0x7ffbec0a2f70 2023-09-30T01:57:53.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0a8f80 con 0x7ffbec0aff90 2023-09-30T01:57:53.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44664/0 (socket says 172.21.15.151:44664) 2023-09-30T01:57:53.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/3924714645 learned_addr learned my addr 172.21.15.151:0/3924714645 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 Processor -- start 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- start start 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 0x7f2c000a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 0x7f2c00148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c000a4ab0 con 0x7f2c0013cf80 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c00146730 con 0x7f2c000acc00 2023-09-30T01:57:53.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c001468a0 con 0x7f2c000acfd0 2023-09-30T01:57:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/3924714645 >> v2:172.21.15.172:3300/0 conn(0x7ffbec0a2f70 msgr2=0x7ffbec0a9680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 --2- 172.21.15.151:0/3924714645 >> v2:172.21.15.172:3300/0 conn(0x7ffbec0a2f70 0x7ffbec0a9680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/3924714645 >> v1:172.21.15.172:6789/0 conn(0x7ffbec0a7ca0 legacy=0x7ffbec13f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/3924714645 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec0a9260 con 0x7ffbec0aff90 2023-09-30T01:57:53.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe9d9b700 1 --2- 172.21.15.151:0/3924714645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0099b0 tx=0x7ffbdc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=589f7f85bc5dd03d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bfffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bfffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50924/0 (socket says 172.21.15.151:50924) 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bfffff700 1 -- 172.21.15.151:0/1202778006 learned_addr learned my addr 172.21.15.151:0/1202778006 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 0x7f2c00148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bff7fe700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 0x7f2c000a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 -- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 msgr2=0x7f2c000a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe959a700 1 -- 172.21.15.151:0/3924714645 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00b760 con 0x7ffbec0aff90 2023-09-30T01:57:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe959a700 1 -- 172.21.15.151:0/3924714645 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc00b8c0 con 0x7ffbec0aff90 2023-09-30T01:57:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbe959a700 1 -- 172.21.15.151:0/3924714645 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00bc80 con 0x7ffbec0aff90 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 0x7f2c000a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 -- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 msgr2=0x7f2c000a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 -- 172.21.15.151:0/1202778006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8008570 con 0x7f2c0013cf80 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bfffff700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:53.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2bff7fe700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 0x7f2c000a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c04ace700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 0x7f2c00148db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c880 tx=0x7f2bf4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4011600 con 0x7f2c0013cf80 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf4011760 con 0x7f2c0013cf80 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- 172.21.15.151:0/1202778006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c000b03c0 con 0x7f2c0013cf80 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4011b20 con 0x7f2c0013cf80 2023-09-30T01:57:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7f2c06531700 1 -- 172.21.15.151:0/1202778006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c00149510 con 0x7f2c0013cf80 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 -- 172.21.15.151:0/3924714645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 msgr2=0x7ffbec0a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.484+0000 7ffbf0b92700 1 --2- 172.21.15.151:0/3924714645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0099b0 tx=0x7ffbdc009250 comp rx=0 tx=0).stop 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 -- 172.21.15.151:0/3924714645 shutdown_connections 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 --2- 172.21.15.151:0/3924714645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0aff90 0x7ffbec0a7760 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0099b0 tx=0x7ffbdc009250 comp rx=0 tx=0).stop 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 --2- 172.21.15.151:0/3924714645 >> v2:172.21.15.172:3300/0 conn(0x7ffbec0a2f70 0x7ffbec0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 -- 172.21.15.151:0/3924714645 >> 172.21.15.151:0/3924714645 conn(0x7ffbec070fe0 msgr2=0x7ffbec0af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 -- 172.21.15.151:0/3924714645 shutdown_connections 2023-09-30T01:57:53.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2c06531700 1 -- 172.21.15.151:0/1202778006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c00067a40 con 0x7f2c0013cf80 2023-09-30T01:57:53.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2bf401a450 con 0x7f2c0013cf80 2023-09-30T01:57:53.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2beffff700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 0x7f2be4062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2bf407c890 con 0x7f2c0013cf80 2023-09-30T01:57:53.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7ffbf0b92700 1 -- 172.21.15.151:0/3924714645 wait complete. 2023-09-30T01:57:53.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2bfffff700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 0x7f2be4062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.488+0000 7f2bfffff700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 0x7f2be4062a80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8006e80 tx=0x7f2bf801b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 Processor -- start 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 -- start start 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 0x7ffbec06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec06c1f0 0x7ffbec154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec140420 con 0x7ffbec06c1f0 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1401e0 con 0x7ffbec0a2f70 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbf0b92700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1415a0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44672/0 (socket says 172.21.15.151:44672) 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/2845756257 learned_addr learned my addr 172.21.15.151:0/2845756257 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbead9d700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec06c1f0 0x7ffbec154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbea59c700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 0x7ffbec06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 msgr2=0x7ffbec06e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 0x7ffbec06e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec06c1f0 msgr2=0x7ffbec154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec06c1f0 0x7ffbec154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 -- 172.21.15.151:0/2845756257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0094f0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbea59c700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 0x7ffbec06e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.492+0000 7ffbe9d9b700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0094c0 tx=0x7ffbdc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00b760 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbf0b92700 1 -- 172.21.15.151:0/2845756257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec13fcc0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbf0b92700 1 -- 172.21.15.151:0/2845756257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec06d840 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c00067a40 con 0x7f2c0013cf80 2023-09-30T01:57:53.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc01de90 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc01a650 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ffbdc01aa80 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 0x7ffbcc062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ffbdc07e380 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.496+0000 7ffbea59c700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 0x7ffbcc062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.500+0000 7ffbf0b92700 1 -- 172.21.15.151:0/2845756257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd00050b0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.500+0000 7ffbea59c700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 0x7ffbcc062bd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0006fd0 tx=0x7ffbe0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.504+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffbdc04ebc0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 Processor -- start 2023-09-30T01:57:53.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 -- start start 2023-09-30T01:57:53.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f53e0140c30 0x7f53e013d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e0141220 con 0x7f53e00ac570 2023-09-30T01:57:53.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f53e0140c30 0x7f53e013d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f53e0140c30 0x7f53e013d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55688/0 (socket says 172.21.15.151:55688) 2023-09-30T01:57:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 -- 172.21.15.151:0/2545400583 learned_addr learned my addr 172.21.15.151:0/2545400583 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e00abdb0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e013de20 con 0x7f53e0140c30 2023-09-30T01:57:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 -- 172.21.15.151:0/2545400583 >> v1:172.21.15.172:6789/0 conn(0x7f53e00ac1a0 legacy=0x7f53e006dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 -- 172.21.15.151:0/2545400583 >> v1:172.21.15.151:6789/0 conn(0x7f53e00ac570 legacy=0x7f53e013d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53dd59a700 1 -- 172.21.15.151:0/2545400583 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151582764 0 0) 0x7f53e0141220 con 0x7f53e00ac570 2023-09-30T01:57:53.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 -- 172.21.15.151:0/2545400583 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e006b0f0 con 0x7f53e0140c30 2023-09-30T01:57:53.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7f53ded9d700 1 --2- 172.21.15.151:0/2545400583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e0140c30 0x7f53e013d8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53e00abf40 tx=0x7f53d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92d42ef105860cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7ff909405700 1 Processor -- start 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.520+0000 7ff909405700 1 -- start start 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff904141220 con 0x7ff9040ac1a0 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040abdb0 con 0x7ff9040ac570 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90413de20 con 0x7ff904140c30 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53dd59a700 1 -- 172.21.15.151:0/2545400583 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d000b860 con 0x7f53e0140c30 2023-09-30T01:57:53.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44688/0 (socket says 172.21.15.151:44688) 2023-09-30T01:57:53.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 -- 172.21.15.151:0/3509070310 learned_addr learned my addr 172.21.15.151:0/3509070310 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53dd59a700 1 -- 172.21.15.151:0/2545400583 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d000b9c0 con 0x7f53e0140c30 2023-09-30T01:57:53.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53dd59a700 1 -- 172.21.15.151:0/2545400583 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d000bd80 con 0x7f53e0140c30 2023-09-30T01:57:53.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 -- 172.21.15.151:0/3509070310 >> v1:172.21.15.172:6789/0 conn(0x7ff9040ac570 legacy=0x7ff90413d1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:57:53.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 -- 172.21.15.151:0/3509070310 >> v1:172.21.15.151:6789/0 conn(0x7ff9040ac1a0 legacy=0x7ff90406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 -- 172.21.15.151:0/3509070310 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90406b0f0 con 0x7ff904140c30 2023-09-30T01:57:53.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff908c04700 1 --2- 172.21.15.151:0/3509070310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009f50 tx=0x7ff8fc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fafe8ccecfaf6e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f2859bcc700 1 Processor -- start 2023-09-30T01:57:53.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff902ffd700 1 -- 172.21.15.151:0/3509070310 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc00e070 con 0x7ff904140c30 2023-09-30T01:57:53.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff902ffd700 1 -- 172.21.15.151:0/3509070310 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc00ba70 con 0x7ff904140c30 2023-09-30T01:57:53.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff902ffd700 1 -- 172.21.15.151:0/3509070310 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc0092e0 con 0x7ff904140c30 2023-09-30T01:57:53.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e0140c30 msgr2=0x7f53e013d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 --2- 172.21.15.151:0/2545400583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e0140c30 0x7f53e013d8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53e00abf40 tx=0x7f53d0009250 comp rx=0 tx=0).stop 2023-09-30T01:57:53.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 shutdown_connections 2023-09-30T01:57:53.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 --2- 172.21.15.151:0/2545400583 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e0140c30 0x7f53e013d8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 >> 172.21.15.151:0/2545400583 conn(0x7f53e00687a0 msgr2=0x7f53e0070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f2859bcc700 1 -- start start 2023-09-30T01:57:53.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f6906ad6700 1 Processor -- start 2023-09-30T01:57:53.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f6906ad6700 1 -- start start 2023-09-30T01:57:53.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- 172.21.15.151:0/3509070310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 msgr2=0x7ff90413d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 --2- 172.21.15.151:0/3509070310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009f50 tx=0x7ff8fc009840 comp rx=0 tx=0).stop 2023-09-30T01:57:53.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- 172.21.15.151:0/3509070310 shutdown_connections 2023-09-30T01:57:53.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 --2- 172.21.15.151:0/3509070310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff90413d8e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- 172.21.15.151:0/3509070310 >> 172.21.15.151:0/3509070310 conn(0x7ff9040687a0 msgr2=0x7ff904070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- 172.21.15.151:0/3509070310 shutdown_connections 2023-09-30T01:57:53.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 shutdown_connections 2023-09-30T01:57:53.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.524+0000 7ff909405700 1 -- 172.21.15.151:0/3509070310 wait complete. 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f285413a370 0x7f285406e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2854136a80 0x7f285413cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285406e5d0 con 0x7f2854071020 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285406e840 con 0x7f2854136a80 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285406ee70 con 0x7f285413a370 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f69000acea0 0x7f690013aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6900141a10 con 0x7f690013b000 2023-09-30T01:57:53.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690006ee30 con 0x7f69000acad0 2023-09-30T01:57:53.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69000ac500 con 0x7f69000acea0 2023-09-30T01:57:53.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2852ffd700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2854071020 0x7f28540acc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41428/0 (socket says 172.21.15.151:41428) 2023-09-30T01:57:53.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2854136a80 0x7f285413cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2852ffd700 1 -- 172.21.15.151:0/1516839851 learned_addr learned my addr 172.21.15.151:0/1516839851 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:53.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f690013b000 0x7f690013d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43364/0 (socket says 172.21.15.151:43364) 2023-09-30T01:57:53.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6904872700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f69000acad0 0x7f69001410d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41444/0 (socket says 172.21.15.151:41444) 2023-09-30T01:57:53.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 -- 172.21.15.151:0/4291859400 learned_addr learned my addr 172.21.15.151:0/4291859400 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:53.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28537fe700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f285413a370 0x7f285406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28527fc700 1 -- 172.21.15.151:0/1516839851 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909881285 0 0) 0x7f285406e5d0 con 0x7f2854071020 2023-09-30T01:57:53.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28527fc700 1 -- 172.21.15.151:0/1516839851 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c003570 con 0x7f2854071020 2023-09-30T01:57:53.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7ff909405700 1 Processor -- start 2023-09-30T01:57:53.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 -- 172.21.15.151:0/2545400583 wait complete. 2023-09-30T01:57:53.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68fffff700 1 --2- 172.21.15.151:0/4291859400 >> v2:172.21.15.172:3300/0 conn(0x7f69000acea0 0x7f690013aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75923586 0 0) 0x7f6900141a10 con 0x7f690013b000 2023-09-30T01:57:53.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68e8003570 con 0x7f690013b000 2023-09-30T01:57:53.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 -- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f285413a370 msgr2=0x7f285406e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f285413a370 0x7f285406e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 -- 172.21.15.151:0/1516839851 >> v1:172.21.15.172:6789/0 conn(0x7f2854071020 legacy=0x7f28540acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 -- 172.21.15.151:0/1516839851 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854136ee0 con 0x7f2854136a80 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457997993 0 0) 0x7f690006ee30 con 0x7f69000acad0 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900141a10 con 0x7f69000acad0 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987612676 0 0) 0x7f68e8003570 con 0x7f690013b000 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 >> v2:172.21.15.172:3300/0 conn(0x7f69000acea0 msgr2=0x7f690013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 --2- 172.21.15.151:0/4291859400 >> v2:172.21.15.172:3300/0 conn(0x7f69000acea0 0x7f690013aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 >> v1:172.21.15.172:6789/0 conn(0x7f69000acad0 legacy=0x7f69001410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690013df40 con 0x7f690013b000 2023-09-30T01:57:53.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f68f80017f0 con 0x7f690013b000 2023-09-30T01:57:53.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2853fff700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2854136a80 0x7f285413cf80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28440098e0 tx=0x7f28440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bd691f2d8cec76 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 >> v1:172.21.15.151:6789/0 conn(0x7f690013b000 legacy=0x7f690013d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e8003bf0 0x7f68e8003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f68e8004810 0x7f68e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 0x7f68e800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69000ac500 con 0x7f68e80051c0 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e800c160 con 0x7f68e8003bf0 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e800c320 con 0x7f68e8004810 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 0x7f68e800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68fffff700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f68e8004810 0x7f68e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6904872700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e8003bf0 0x7f68e8003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 Processor -- start 2023-09-30T01:57:53.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28527fc700 1 -- 172.21.15.151:0/1516839851 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844011630 con 0x7f2854136a80 2023-09-30T01:57:53.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 -- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f68e8004810 msgr2=0x7f68e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f68e8004810 0x7f68e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 -- 172.21.15.151:0/4291859400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e8003bf0 msgr2=0x7f68e8003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e8003bf0 0x7f68e8003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 -- 172.21.15.151:0/4291859400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e8003570 con 0x7f68e80051c0 2023-09-30T01:57:53.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 -- 172.21.15.151:0/4291859400 reap_dead start 2023-09-30T01:57:53.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28527fc700 1 -- 172.21.15.151:0/1516839851 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844011790 con 0x7f2854136a80 2023-09-30T01:57:53.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f28527fc700 1 -- 172.21.15.151:0/1516839851 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844011b30 con 0x7f2854136a80 2023-09-30T01:57:53.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 -- start start 2023-09-30T01:57:53.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6905073700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 0x7f68e800ba60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f68f800a8a0 tx=0x7f68f8002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8ff00d3574646e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f80070d0 con 0x7f68e80051c0 2023-09-30T01:57:53.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f8007230 con 0x7f68e80051c0 2023-09-30T01:57:53.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f68ff7fe700 1 -- 172.21.15.151:0/4291859400 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f80075d0 con 0x7f68e80051c0 2023-09-30T01:57:53.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2854136a80 msgr2=0x7f285413cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2854136a80 0x7f285413cf80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28440098e0 tx=0x7f28440091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- 172.21.15.151:0/1516839851 shutdown_connections 2023-09-30T01:57:53.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2854136a80 0x7f285413cf80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 --2- 172.21.15.151:0/1516839851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f285413a370 0x7f285406e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f2859bcc700 1 -- 172.21.15.151:0/1516839851 >> 172.21.15.151:0/1516839851 conn(0x7f2854070510 msgr2=0x7f2854070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e00ac570 0x7f53e013fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53e0140c30 0x7f53e00a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00a6880 con 0x7f53e00ac570 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00a8340 con 0x7f53e0140c30 2023-09-30T01:57:53.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53df59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00a85b0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53de59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53de59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44722/0 (socket says 172.21.15.151:44722) 2023-09-30T01:57:53.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53de59c700 1 -- 172.21.15.151:0/738526420 learned_addr learned my addr 172.21.15.151:0/738526420 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- 172.21.15.151:0/1516839851 shutdown_connections 2023-09-30T01:57:53.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7ff909405700 1 -- start start 2023-09-30T01:57:53.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f53ddd9b700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e00ac570 0x7f53e013fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 -- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53e0140c30 msgr2=0x7f53e00a6d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:57:53.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53e0140c30 0x7f53e00a6d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 -- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e00ac570 msgr2=0x7f53e013fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e00ac570 0x7f53e013fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 -- 172.21.15.151:0/738526420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00094f0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 -- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 msgr2=0x7f68e800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.528+0000 7f6906ad6700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 0x7f68e800ba60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f68f800a8a0 tx=0x7f68f8002190 comp rx=0 tx=0).stop 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- 172.21.15.151:0/4291859400 shutdown_connections 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 --2- 172.21.15.151:0/4291859400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68e80051c0 0x7f68e800ba60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- 172.21.15.151:0/4291859400 >> 172.21.15.151:0/4291859400 conn(0x7f6900070510 msgr2=0x7f6900070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- 172.21.15.151:0/4291859400 shutdown_connections 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- 172.21.15.151:0/4291859400 wait complete. 2023-09-30T01:57:53.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 Processor -- start 2023-09-30T01:57:53.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 0x7ff904117620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 -- start start 2023-09-30T01:57:53.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7fd70a9c7700 1 Processor -- start 2023-09-30T01:57:53.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7fd70a9c7700 1 -- start start 2023-09-30T01:57:53.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- 172.21.15.151:0/1516839851 wait complete. 2023-09-30T01:57:53.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 Processor -- start 2023-09-30T01:57:53.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- start start 2023-09-30T01:57:53.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 Processor -- start 2023-09-30T01:57:53.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- start start 2023-09-30T01:57:53.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53de59c700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53d800cb40 tx=0x7f53d8001760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff904141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff904144140 con 0x7ff9040ac570 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff904119080 con 0x7ff9040ac1a0 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9041432d0 con 0x7ff904140c30 2023-09-30T01:57:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff9037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff9037fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55730/0 (socket says 172.21.15.151:55730) 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff9037fe700 1 -- 172.21.15.151:0/4135536321 learned_addr learned my addr 172.21.15.151:0/4135536321 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff904141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff903fff700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 0x7ff904117620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 -- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 msgr2=0x7ff904117620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 0x7ff904117620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 -- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 msgr2=0x7ff9041418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 -- 172.21.15.151:0/4135536321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc009ae0 con 0x7ff904140c30 2023-09-30T01:57:53.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff9037fe700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff903fff700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 0x7ff904117620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff908c04700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff904141de0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff904070c90 tx=0x7ff8fc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 0x7f7ca40a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7ca40abf10 0x7f7ca40a6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca40a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca413b130 con 0x7f7ca40abf10 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca40b0590 con 0x7f7ca40b01c0 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca9664700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca40a8480 con 0x7f7ca40a96c0 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2854071020 0x7f28540af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2854136a80 0x7f28540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540a6630 con 0x7f285413a370 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540a68e0 con 0x7f2854136a80 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2859bcc700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540a5e40 con 0x7f2854071020 2023-09-30T01:57:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55756/0 (socket says 172.21.15.151:55756) 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 -- 172.21.15.151:0/2145424994 learned_addr learned my addr 172.21.15.151:0/2145424994 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2852ffd700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2854136a80 0x7f28540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 -- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2854071020 msgr2=0x7f28540af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2854071020 0x7f28540af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 -- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2854136a80 msgr2=0x7f28540a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2854136a80 0x7f28540a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 -- 172.21.15.151:0/2145424994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844009420 con 0x7f285413a370 2023-09-30T01:57:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f2853fff700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28540ad630 tx=0x7f2844017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc0094e0 con 0x7ff904140c30 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc012db0 con 0x7ff904140c30 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff904139e10 con 0x7ff904140c30 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9041445d0 con 0x7ff904140c30 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc01d650 con 0x7ff904140c30 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d800a7a0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d800a940 con 0x7f53e00ac1a0 2023-09-30T01:57:53.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d800ad00 con 0x7f53e00ac1a0 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0140290 con 0x7f53e00ac1a0 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e00a7530 con 0x7f53e00ac1a0 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e80051c0 0x7f6900140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6900150a50 0x7f6900150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6900150600 con 0x7f69000adf10 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6900118660 con 0x7f68e80051c0 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6906ad6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6900151400 con 0x7f6900150a50 2023-09-30T01:57:53.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55768/0 (socket says 172.21.15.151:55768) 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 -- 172.21.15.151:0/1520288313 learned_addr learned my addr 172.21.15.151:0/1520288313 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f6905073700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6900150a50 0x7f6900150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 -- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6900150a50 msgr2=0x7f6900150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6900150a50 0x7f6900150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 -- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e80051c0 msgr2=0x7f6900140320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e80051c0 0x7f6900140320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f68fffff700 1 -- 172.21.15.151:0/1520288313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000af650 con 0x7f69000adf10 2023-09-30T01:57:53.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca2ffd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 0x7f7ca40a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca37fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca40a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca37fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca40a6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44744/0 (socket says 172.21.15.151:44744) 2023-09-30T01:57:53.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca37fe700 1 -- 172.21.15.151:0/119877650 learned_addr learned my addr 172.21.15.151:0/119877650 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 --2- 172.21.15.151:0/119877650 >> v2:172.21.15.151:3300/0 conn(0x7f7ca40abf10 0x7f7ca40a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f68fffff700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f68f4011b10 tx=0x7f68f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 -- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 msgr2=0x7f7ca40a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 --2- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca40a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 -- 172.21.15.151:0/119877650 >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 msgr2=0x7f7ca40a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 --2- 172.21.15.151:0/119877650 >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 0x7f7ca40a8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.532+0000 7f7ca27fc700 1 -- 172.21.15.151:0/119877650 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40a91a0 con 0x7f7ca40abf10 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca2ffd700 1 --2- 172.21.15.151:0/119877650 >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 0x7f7ca40a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca27fc700 1 --2- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40abf10 0x7f7ca40a6930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ad3d0 tx=0x7f7c980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5242c60bf2a6be72 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400f990 con 0x7f69000adf10 2023-09-30T01:57:53.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6900137ce0 con 0x7f69000adf10 2023-09-30T01:57:53.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69001381b0 con 0x7f69000adf10 2023-09-30T01:57:53.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca1ffb700 1 -- 172.21.15.151:0/119877650 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98011630 con 0x7f7ca40abf10 2023-09-30T01:57:53.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca1ffb700 1 -- 172.21.15.151:0/119877650 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c980117b0 con 0x7f7ca40abf10 2023-09-30T01:57:53.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca1ffb700 1 -- 172.21.15.151:0/119877650 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98011b70 con 0x7f7ca40abf10 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd70a9c7700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd7040ac810 0x7fd704145010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd70a9c7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd70406b990 0x7fd70413e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd70a9c7700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd704145ad0 con 0x7fd704145550 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd70a9c7700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7040acbe0 con 0x7fd7040ac810 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd70a9c7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70406bdf0 con 0x7fd70406b990 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd7040ac810 0x7fd704145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e00a69f0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd7040ac810 0x7fd704145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55770/0 (socket says 172.21.15.151:55770) 2023-09-30T01:57:53.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 -- 172.21.15.151:0/1315962215 learned_addr learned my addr 172.21.15.151:0/1315962215 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f400faf0 con 0x7f69000adf10 2023-09-30T01:57:53.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400fe70 con 0x7f69000adf10 2023-09-30T01:57:53.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28440233f0 con 0x7f285413a370 2023-09-30T01:57:53.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e40050b0 con 0x7ff904140c30 2023-09-30T01:57:53.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690013b580 con 0x7f69000adf10 2023-09-30T01:57:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 -- 172.21.15.151:0/1315962215 >> v2:172.21.15.172:3300/0 conn(0x7fd70406b990 msgr2=0x7fd70413e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 --2- 172.21.15.151:0/1315962215 >> v2:172.21.15.172:3300/0 conn(0x7fd70406b990 0x7fd70413e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 -- 172.21.15.151:0/1315962215 >> v1:172.21.15.172:6789/0 conn(0x7fd704145550 legacy=0x7fd70406b280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:57:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 -- 172.21.15.151:0/1315962215 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70406d290 con 0x7fd7040ac810 2023-09-30T01:57:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff8fc01da80 con 0x7ff904140c30 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 -- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40abf10 msgr2=0x7f7ca40a6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 --2- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40abf10 0x7f7ca40a6930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ad3d0 tx=0x7f7c980091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844023550 con 0x7f285413a370 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540b04f0 con 0x7f285413a370 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844023910 con 0x7f285413a370 2023-09-30T01:57:53.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285415d730 con 0x7f285413a370 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7ff8ebfff700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 0x7ff8f0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff8fc07ddd0 con 0x7ff904140c30 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 -- 172.21.15.151:0/119877650 shutdown_connections 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 --2- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca40a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 --2- 172.21.15.151:0/119877650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40abf10 0x7f7ca40a6930 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 --2- 172.21.15.151:0/119877650 >> v2:172.21.15.172:3300/0 conn(0x7f7ca40b01c0 0x7f7ca40a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f7ca9664700 1 -- 172.21.15.151:0/119877650 >> 172.21.15.151:0/119877650 conn(0x7f7ca4070fe0 msgr2=0x7f7ca40af880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f53d8003030 con 0x7f53e00ac1a0 2023-09-30T01:57:53.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7fd703fff700 1 --2- 172.21.15.151:0/1315962215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd704145010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd704145a40 tx=0x7fd6f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b8b38443421ca3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.536+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f68f4014650 con 0x7f69000adf10 2023-09-30T01:57:53.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7ff903fff700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 0x7ff8f0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f53ce7fc700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 0x7f53c0062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f53d807cdb0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- 172.21.15.151:0/119877650 shutdown_connections 2023-09-30T01:57:53.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- 172.21.15.151:0/119877650 wait complete. 2023-09-30T01:57:53.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f284402c3b0 con 0x7f285413a370 2023-09-30T01:57:53.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f68fcff9700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 0x7f68ec062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f68f4018030 con 0x7f69000adf10 2023-09-30T01:57:53.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f283b7fe700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 0x7f2840062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f6904872700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 0x7f68ec062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f284407df70 con 0x7f285413a370 2023-09-30T01:57:53.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f53ddd9b700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 0x7f53c0062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f28537fe700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 0x7f2840062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f6904872700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 0x7f68ec062b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68f0007e90 tx=0x7f68f0007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 Processor -- start 2023-09-30T01:57:53.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- start start 2023-09-30T01:57:53.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f28537fe700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 0x7f2840062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2848009b10 tx=0x7f28480091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f53ddd9b700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 0x7f53c0062ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53e00a5f50 tx=0x7f53d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca4140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca40b01c0 0x7f7ca414fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 0x7f7ca40a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca40a5b80 con 0x7f7ca40a4d40 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca414a2a0 con 0x7f7ca40b01c0 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca9664700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca41416d0 con 0x7f7ca40a96c0 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca2ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca4140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 0x7f7ca40a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca2ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca4140a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44766/0 (socket says 172.21.15.151:44766) 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca2ffd700 1 -- 172.21.15.151:0/3380583406 learned_addr learned my addr 172.21.15.151:0/3380583406 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca27fc700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca40b01c0 0x7f7ca414fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 -- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 msgr2=0x7f7ca4140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca4140a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 -- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca40b01c0 msgr2=0x7f7ca414fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca40b01c0 0x7f7ca414fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 -- 172.21.15.151:0/3380583406 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c009a20 con 0x7f7ca40a4d40 2023-09-30T01:57:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.540+0000 7f7ca37fe700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 0x7f7ca40a3380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40a9ab0 tx=0x7f7c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28340050b0 con 0x7f285413a370 2023-09-30T01:57:53.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7ff903fff700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 0x7ff8f0062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4007fd0 tx=0x7ff8f4007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7fd702ffd700 1 -- 172.21.15.151:0/1315962215 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f400b820 con 0x7fd7040ac810 2023-09-30T01:57:53.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f690013b580 con 0x7f69000adf10 2023-09-30T01:57:53.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94013680 con 0x7f7ca40a4d40 2023-09-30T01:57:53.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c98009420 con 0x7f7ca40a4d40 2023-09-30T01:57:53.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca4150450 con 0x7f7ca40a4d40 2023-09-30T01:57:53.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.544+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f53d804d6f0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.548+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f284404e7b0 con 0x7f285413a370 2023-09-30T01:57:53.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7fd702ffd700 1 -- 172.21.15.151:0/1315962215 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f400b980 con 0x7fd7040ac810 2023-09-30T01:57:53.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7fd702ffd700 1 -- 172.21.15.151:0/1315962215 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f400bd20 con 0x7fd7040ac810 2023-09-30T01:57:53.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff8fc081030 con 0x7ff904140c30 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1315962215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 msgr2=0x7fd704145010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1315962215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd704145010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd704145a40 tx=0x7fd6f4009250 comp rx=0 tx=0).stop 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94005a00 con 0x7f7ca40a4d40 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94005b70 con 0x7f7ca40a4d40 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7c94005d10 con 0x7f7ca40a4d40 2023-09-30T01:57:53.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7c8affd700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 0x7f7c8c062b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7c9407c330 con 0x7f7ca40a4d40 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7ca2ffd700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 0x7f7c8c062b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.552+0000 7f7ca2ffd700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 0x7f7c8c062b80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00f750 tx=0x7f7c9c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.556+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c840050b0 con 0x7f7ca40a4d40 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1315962215 shutdown_connections 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1315962215 >> v2:172.21.15.172:3300/0 conn(0x7fd70406b990 0x7fd70413e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1315962215 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd704145010 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1315962215 >> 172.21.15.151:0/1315962215 conn(0x7fd704070510 msgr2=0x7fd704070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c9407ae60 con 0x7f7ca40a4d40 2023-09-30T01:57:53.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.564+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1315962215 shutdown_connections 2023-09-30T01:57:53.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.568+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1315962215 wait complete. 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.568+0000 7fd70a9c7700 1 Processor -- start 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 -- start start 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd7041448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 0x7fd7041553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704155c30 con 0x7fd7040ac810 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704155930 con 0x7fd70406b990 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.572+0000 7fd70a9c7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70413b620 con 0x7fd704145550 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.151:50988/0 (socket says 172.21.15.151:50988) 2023-09-30T01:57:53.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 -- 172.21.15.151:0/1601722306 learned_addr learned my addr 172.21.15.151:0/1601722306 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd708f64700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 0x7fd7041553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 -- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 msgr2=0x7fd7041553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 0x7fd7041553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 -- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 msgr2=0x7fd7041448a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd7041448a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 -- 172.21.15.151:0/1601722306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f40094f0 con 0x7fd70406b990 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd708f64700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 0x7fd7041553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd703fff700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4003080 tx=0x7fd6f4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f401b470 con 0x7fd70406b990 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70413b840 con 0x7fd70406b990 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70413bd80 con 0x7fd70406b990 2023-09-30T01:57:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f401b5d0 con 0x7fd70406b990 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f401b970 con 0x7fd70406b990 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.576+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd6f401bce0 con 0x7fd70406b990 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.580+0000 7fd6ebfff700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 0x7fd6ec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.580+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd6f407e750 con 0x7fd70406b990 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.580+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e40050b0 con 0x7fd70406b990 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.580+0000 7fd7037fe700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 0x7fd6ec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:53.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.580+0000 7fd7037fe700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 0x7fd6ec062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7040a5760 tx=0x7fd6f8006bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:53.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.592+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd6f402a020 con 0x7fd70406b990 2023-09-30T01:57:53.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.784+0000 7f2c06531700 1 -- 172.21.15.151:0/1202778006 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2c000670a0 con 0x7f2c0013cf80 2023-09-30T01:57:53.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.788+0000 7f2beffff700 1 -- 172.21.15.151:0/1202778006 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2bf404cfd0 con 0x7f2c0013cf80 2023-09-30T01:57:53.794 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 msgr2=0x7f2be4062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 0x7f2be4062a80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8006e80 tx=0x7f2bf801b000 comp rx=0 tx=0).stop 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 msgr2=0x7f2c00148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 0x7f2c00148db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c880 tx=0x7f2bf4008040 comp rx=0 tx=0).stop 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 shutdown_connections 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2be40605d0 0x7f2be4062a80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2c0013cf80 0x7f2c00148db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2c000acfd0 0x7f2c000a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 --2- 172.21.15.151:0/1202778006 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c000acc00 0x7f2c000a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 >> 172.21.15.151:0/1202778006 conn(0x7f2c00070510 msgr2=0x7f2c00138e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 shutdown_connections 2023-09-30T01:57:53.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.792+0000 7f2bedffb700 1 -- 172.21.15.151:0/1202778006 wait complete. 2023-09-30T01:57:53.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbf0b92700 1 -- 172.21.15.151:0/2845756257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffbd0005360 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.809 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:53.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbda7fc700 1 -- 172.21.15.151:0/2845756257 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffbdc0521e0 con 0x7ffbec0a7ca0 2023-09-30T01:57:53.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.6 2023-09-30T01:57:53.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 msgr2=0x7ffbcc062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 0x7ffbcc062bd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0006fd0 tx=0x7ffbe0008040 comp rx=0 tx=0).stop 2023-09-30T01:57:53.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 msgr2=0x7ffbec06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.804+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0094c0 tx=0x7ffbdc016000 comp rx=0 tx=0).stop 2023-09-30T01:57:53.812 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 shutdown_connections 2023-09-30T01:57:53.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ffbcc060720 0x7ffbcc062bd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbec06c1f0 0x7ffbec154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbec0a7ca0 0x7ffbec06bcb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 --2- 172.21.15.151:0/2845756257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbec0a2f70 0x7ffbec06e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 >> 172.21.15.151:0/2845756257 conn(0x7ffbec070fe0 msgr2=0x7ffbec0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 shutdown_connections 2023-09-30T01:57:53.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.808+0000 7ffbcbfff700 1 -- 172.21.15.151:0/2845756257 wait complete. 2023-09-30T01:57:53.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2023-09-30T01:57:53.825 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff8e4005360 con 0x7ff904140c30 2023-09-30T01:57:53.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff8ebfff700 1 -- 172.21.15.151:0/4135536321 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8fc04e590 con 0x7ff904140c30 2023-09-30T01:57:53.841 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:53.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f690013b580 con 0x7f69000adf10 2023-09-30T01:57:53.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7f68fcff9700 1 -- 172.21.15.151:0/1520288313 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f690013b580 con 0x7f69000adf10 2023-09-30T01:57:53.842 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:53.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 msgr2=0x7ff8f0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 0x7ff8f0062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4007fd0 tx=0x7ff8f4007e70 comp rx=0 tx=0).stop 2023-09-30T01:57:53.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 msgr2=0x7ff904141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff904141de0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff904070c90 tx=0x7ff8fc017000 comp rx=0 tx=0).stop 2023-09-30T01:57:53.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 shutdown_connections 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff8f00606a0 0x7ff8f0062b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff904140c30 0x7ff904141de0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9040ac570 0x7ff9041418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 --2- 172.21.15.151:0/4135536321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9040ac1a0 0x7ff904117620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 >> 172.21.15.151:0/4135536321 conn(0x7ff9040687a0 msgr2=0x7ff90406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 shutdown_connections 2023-09-30T01:57:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.836+0000 7ff909405700 1 -- 172.21.15.151:0/4135536321 wait complete. 2023-09-30T01:57:53.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 msgr2=0x7f68ec062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 0x7f68ec062b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68f0007e90 tx=0x7f68f0007d60 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 msgr2=0x7f69000ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f68f4011b10 tx=0x7f68f40091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 shutdown_connections 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f68ec060650 0x7f68ec062b00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6900150a50 0x7f6900150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69000adf10 0x7f69000ae380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 --2- 172.21.15.151:0/1520288313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68e80051c0 0x7f6900140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 >> 172.21.15.151:0/1520288313 conn(0x7f6900070510 msgr2=0x7f6900141040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 shutdown_connections 2023-09-30T01:57:53.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.840+0000 7f6906ad6700 1 -- 172.21.15.151:0/1520288313 wait complete. 2023-09-30T01:57:53.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.844+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f53e00008d0 con 0x7f53e00ac1a0 2023-09-30T01:57:53.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.844+0000 7f53ce7fc700 1 -- 172.21.15.151:0/738526420 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f53d800ae60 con 0x7f53e00ac1a0 2023-09-30T01:57:53.851 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:53.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.4 2023-09-30T01:57:53.854 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 msgr2=0x7f53c0062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 0x7f53c0062ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53e00a5f50 tx=0x7f53d4009040 comp rx=0 tx=0).stop 2023-09-30T01:57:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 msgr2=0x7f53e00a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53d800cb40 tx=0x7f53d8001760 comp rx=0 tx=0).stop 2023-09-30T01:57:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 shutdown_connections 2023-09-30T01:57:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f53c0060620 0x7f53c0062ad0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53e0140c30 0x7f53e00a6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53e00ac570 0x7f53e013fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 --2- 172.21.15.151:0/738526420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53e00ac1a0 0x7f53e00a61f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 >> 172.21.15.151:0/738526420 conn(0x7f53e00687a0 msgr2=0x7f53e013c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2834005360 con 0x7f285413a370 2023-09-30T01:57:53.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 shutdown_connections 2023-09-30T01:57:53.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f53df59e700 1 -- 172.21.15.151:0/738526420 wait complete. 2023-09-30T01:57:53.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.848+0000 7f283b7fe700 1 -- 172.21.15.151:0/2145424994 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2844028030 con 0x7f285413a370 2023-09-30T01:57:53.858 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-09-30T01:57:53.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 msgr2=0x7f2840062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 0x7f2840062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2848009b10 tx=0x7f28480091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 msgr2=0x7f285415d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28540ad630 tx=0x7f2844017000 comp rx=0 tx=0).stop 2023-09-30T01:57:53.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 shutdown_connections 2023-09-30T01:57:53.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f28400606a0 0x7f2840062b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f285413a370 0x7f285415d1f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2854136a80 0x7f28540a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 --2- 172.21.15.151:0/2145424994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2854071020 0x7f28540af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 >> 172.21.15.151:0/2145424994 conn(0x7f2854070510 msgr2=0x7f28541375c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 shutdown_connections 2023-09-30T01:57:53.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.852+0000 7f2859bcc700 1 -- 172.21.15.151:0/2145424994 wait complete. 2023-09-30T01:57:53.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2023-09-30T01:57:53.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.856+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7c84005630 con 0x7f7ca40a4d40 2023-09-30T01:57:53.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.856+0000 7f7c8affd700 1 -- 172.21.15.151:0/3380583406 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c94021070 con 0x7f7ca40a4d40 2023-09-30T01:57:53.863 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.863 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:53.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 msgr2=0x7f7c8c062b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 0x7f7c8c062b80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00f750 tx=0x7f7c9c00b040 comp rx=0 tx=0).stop 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 msgr2=0x7f7ca40a3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 0x7f7ca40a3380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40a9ab0 tx=0x7f7c94009250 comp rx=0 tx=0).stop 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 shutdown_connections 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7c8c0606d0 0x7f7c8c062b80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ca40a4d40 0x7f7ca40a3380 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca40b01c0 0x7f7ca414fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 --2- 172.21.15.151:0/3380583406 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ca40a96c0 0x7f7ca4140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 >> 172.21.15.151:0/3380583406 conn(0x7f7ca4070fe0 msgr2=0x7f7ca40af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 shutdown_connections 2023-09-30T01:57:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7f7ca9664700 1 -- 172.21.15.151:0/3380583406 wait complete. 2023-09-30T01:57:53.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd6e4005360 con 0x7fd70406b990 2023-09-30T01:57:53.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.860+0000 7fd6ebfff700 1 -- 172.21.15.151:0/1601722306 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6f404ee90 con 0x7fd70406b990 2023-09-30T01:57:53.867 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-09-30T01:57:53.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 msgr2=0x7fd6ec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 0x7fd6ec062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7040a5760 tx=0x7fd6f8006bd0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 msgr2=0x7fd7040a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4003080 tx=0x7fd6f4016000 comp rx=0 tx=0).stop 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 shutdown_connections 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd6ec060670 0x7fd6ec062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd704145550 0x7fd7041553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd7040ac810 0x7fd7041448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 --2- 172.21.15.151:0/1601722306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd70406b990 0x7fd7040a4440 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 >> 172.21.15.151:0/1601722306 conn(0x7fd704070510 msgr2=0x7fd7040aad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-30T01:57:53.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 shutdown_connections 2023-09-30T01:57:53.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:53.864+0000 7fd70a9c7700 1 -- 172.21.15.151:0/1601722306 wait complete. 2023-09-30T01:57:53.871 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2023-09-30T01:57:53.872 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-09-30T01:57:53.875 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.879 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.7 2023-09-30T01:57:53.879 DEBUG:teuthology.parallel:result is None 2023-09-30T01:57:53.879 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:57:53.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:57:54.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 Processor -- start 2023-09-30T01:57:54.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 -- start start 2023-09-30T01:57:54.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd2406b550 0x7efd240ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2406c8e0 con 0x7efd240ab460 2023-09-30T01:57:54.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240abe70 con 0x7efd2406b180 2023-09-30T01:57:54.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.016+0000 7efd2884d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd24141ce0 con 0x7efd2406b550 2023-09-30T01:57:54.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2259c700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efd240ab460 0x7efd2413e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43368/0 (socket says 172.21.15.151:43368) 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2259c700 1 -- 172.21.15.151:0/2757149701 learned_addr learned my addr 172.21.15.151:0/2757149701 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2359e700 1 --2- 172.21.15.151:0/2757149701 >> v2:172.21.15.172:3300/0 conn(0x7efd2406b550 0x7efd240ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000678879 0 0) 0x7efd2406c8e0 con 0x7efd240ab460 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10003570 con 0x7efd240ab460 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4070804108 0 0) 0x7efd10003570 con 0x7efd240ab460 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 >> v2:172.21.15.172:3300/0 conn(0x7efd2406b550 msgr2=0x7efd240ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 --2- 172.21.15.151:0/2757149701 >> v2:172.21.15.172:3300/0 conn(0x7efd2406b550 0x7efd240ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 >> v1:172.21.15.172:6789/0 conn(0x7efd2406b180 legacy=0x7efd241415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd240a9b00 con 0x7efd240ab460 2023-09-30T01:57:54.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7efd0c001470 con 0x7efd240ab460 2023-09-30T01:57:54.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2359e700 1 --2- 172.21.15.151:0/2757149701 >> v2:172.21.15.172:3300/0 conn(0x7efd2406b550 0x7efd240ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:54.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 >> v1:172.21.15.151:6789/0 conn(0x7efd240ab460 legacy=0x7efd2413e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 0x7efd10003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd10004810 0x7efd10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd100052e0 0x7efd1000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd24141ce0 con 0x7efd100052e0 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1000c2a0 con 0x7efd10004810 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1000c460 con 0x7efd10003a60 2023-09-30T01:57:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 0x7efd10003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2359e700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd100052e0 0x7efd1000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2259c700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd10004810 0x7efd10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 -- 172.21.15.151:0/2757149701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd10004810 msgr2=0x7efd10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd10004810 0x7efd10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 -- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd100052e0 msgr2=0x7efd1000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd100052e0 0x7efd1000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2359e700 1 -- 172.21.15.151:0/2757149701 reap_dead start 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 -- 172.21.15.151:0/2757149701 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10003570 con 0x7efd10003a60 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd22d9d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 0x7efd10003e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd140018b0 tx=0x7efd14007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b1984bae45fa83 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14012420 con 0x7efd10003a60 2023-09-30T01:57:54.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd140125a0 con 0x7efd10003a60 2023-09-30T01:57:54.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd21d9b700 1 -- 172.21.15.151:0/2757149701 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14012960 con 0x7efd10003a60 2023-09-30T01:57:54.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 -- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 msgr2=0x7efd10003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 0x7efd10003e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd140018b0 tx=0x7efd14007d30 comp rx=0 tx=0).stop 2023-09-30T01:57:54.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 -- 172.21.15.151:0/2757149701 shutdown_connections 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 --2- 172.21.15.151:0/2757149701 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd10003a60 0x7efd10003e70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 -- 172.21.15.151:0/2757149701 >> 172.21.15.151:0/2757149701 conn(0x7efd24070510 msgr2=0x7efd24070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 -- 172.21.15.151:0/2757149701 shutdown_connections 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.020+0000 7efd2884d700 1 -- 172.21.15.151:0/2757149701 wait complete. 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 Processor -- start 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 -- start start 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd24150e60 0x7efd24151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd241517b0 0x7efd24154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd24145360 con 0x7efd10003a60 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd24143300 con 0x7efd24150e60 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2884d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2414e8a0 con 0x7efd241517b0 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55806/0 (socket says 172.21.15.151:55806) 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 -- 172.21.15.151:0/294108535 learned_addr learned my addr 172.21.15.151:0/294108535 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2359e700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd241517b0 0x7efd24154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd2259c700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd24150e60 0x7efd24151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 -- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd241517b0 msgr2=0x7efd24154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd241517b0 0x7efd24154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 -- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd24150e60 msgr2=0x7efd24151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd24150e60 0x7efd24151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.024+0000 7efd22d9d700 1 -- 172.21.15.151:0/294108535 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24145560 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd22d9d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c3f0 tx=0x7efd14015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14001200 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1400ce80 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2406e920 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14012940 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd241547c0 con 0x7efd10003a60 2023-09-30T01:57:54.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd24151b80 con 0x7efd10003a60 2023-09-30T01:57:54.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.028+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efd1401f620 con 0x7efd10003a60 2023-09-30T01:57:54.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.032+0000 7efd1affd700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 0x7efd04062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.032+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efd1407e200 con 0x7efd10003a60 2023-09-30T01:57:54.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.032+0000 7efd2259c700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 0x7efd04062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.032+0000 7efd2259c700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 0x7efd04062a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0015a0 tx=0x7efd0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.036+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd1404e9e0 con 0x7efd10003a60 2023-09-30T01:57:54.190 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.184+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd241430b0 con 0x7efd04060580 2023-09-30T01:57:54.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.188+0000 7efd1affd700 1 -- 172.21.15.151:0/294108535 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+45866 (secure 0 0 0) 0x7efd241430b0 con 0x7efd04060580 2023-09-30T01:57:54.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:54.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 msgr2=0x7efd04062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 0x7efd04062a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0015a0 tx=0x7efd0c009040 comp rx=0 tx=0).stop 2023-09-30T01:57:54.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 msgr2=0x7efd24143850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c3f0 tx=0x7efd14015000 comp rx=0 tx=0).stop 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 shutdown_connections 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efd04060580 0x7efd04062a30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd241517b0 0x7efd24154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd24150e60 0x7efd24151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 --2- 172.21.15.151:0/294108535 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd10003a60 0x7efd24143850 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 >> 172.21.15.151:0/294108535 conn(0x7efd24070510 msgr2=0x7efd240aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 shutdown_connections 2023-09-30T01:57:54.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.192+0000 7efd2884d700 1 -- 172.21.15.151:0/294108535 wait complete. 2023-09-30T01:57:54.199 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T01:57:54.208 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-09-30T01:57:53.435688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42976,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931744,"statfs":{"total":773094113280,"available":773050105856,"internally_reserved":0,"allocated":2392064,"data_stored":1474082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.880316"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.544219+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:50.544219+0000","last_peered":"2023-09-30T01:57:50.544219+0000","last_clean":"2023-09-30T01:57:50.544219+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:50.544219+0000","last_undegraded":"2023-09-30T01:57:50.544219+0000","last_fullsized":"2023-09-30T01:57:50.544219+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:18:26.843921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.109930+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.109930+0000","last_peered":"2023-09-30T01:57:51.109930+0000","last_clean":"2023-09-30T01:57:51.109930+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.109930+0000","last_undegraded":"2023-09-30T01:57:51.109930+0000","last_fullsized":"2023-09-30T01:57:51.109930+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111284+0000","last_change":"2023-09-30T01:57:51.106990+0000","last_active":"2023-09-30T01:57:51.111284+0000","last_peered":"2023-09-30T01:57:51.111284+0000","last_clean":"2023-09-30T01:57:51.111284+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T01:57:51.111284+0000","last_undegraded":"2023-09-30T01:57:51.111284+0000","last_fullsized":"2023-09-30T01:57:51.111284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:40.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.547452+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.547452+0000","last_peered":"2023-09-30T01:57:50.547452+0000","last_clean":"2023-09-30T01:57:50.547452+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.547452+0000","last_undegraded":"2023-09-30T01:57:50.547452+0000","last_fullsized":"2023-09-30T01:57:50.547452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.701554+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.701554+0000","last_peered":"2023-09-30T01:57:50.701554+0000","last_clean":"2023-09-30T01:57:50.701554+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.701554+0000","last_undegraded":"2023-09-30T01:57:50.701554+0000","last_fullsized":"2023-09-30T01:57:50.701554+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.547375+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.547375+0000","last_peered":"2023-09-30T01:57:50.547375+0000","last_clean":"2023-09-30T01:57:50.547375+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.547375+0000","last_undegraded":"2023-09-30T01:57:50.547375+0000","last_fullsized":"2023-09-30T01:57:50.547375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549603+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.549603+0000","last_peered":"2023-09-30T01:57:50.549603+0000","last_clean":"2023-09-30T01:57:50.549603+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.549603+0000","last_undegraded":"2023-09-30T01:57:50.549603+0000","last_fullsized":"2023-09-30T01:57:50.549603+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.126241+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.126241+0000","last_peered":"2023-09-30T01:57:51.126241+0000","last_clean":"2023-09-30T01:57:51.126241+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.126241+0000","last_undegraded":"2023-09-30T01:57:51.126241+0000","last_fullsized":"2023-09-30T01:57:51.126241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365712,"statfs":{"total":96636764160,"available":96630489088,"internally_reserved":0,"allocated":696320,"data_stored":580579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2829999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5156,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366684,"statfs":{"total":96636764160,"available":96631484416,"internally_reserved":0,"allocated":159744,"data_stored":50316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5164,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366676,"statfs":{"total":96636764160,"available":96631476224,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365712,"statfs":{"total":96636764160,"available":96630489088,"internally_reserved":0,"allocated":696320,"data_stored":580192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0190000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:57:54.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:57:54.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 Processor -- start 2023-09-30T01:57:54.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- start start 2023-09-30T01:57:54.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9e04140ae0 0x7f9e04140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e040b41b0 con 0x7f9e0406b350 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0406c790 con 0x7f9e04140ae0 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0406c990 con 0x7f9e040b3de0 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44788/0 (socket says 172.21.15.151:44788) 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 -- 172.21.15.151:0/3277681661 learned_addr learned my addr 172.21.15.151:0/3277681661 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0aa16700 1 --2- 172.21.15.151:0/3277681661 >> v2:172.21.15.151:3300/0 conn(0x7f9e04140ae0 0x7f9e04140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 -- 172.21.15.151:0/3277681661 >> v2:172.21.15.151:3300/0 conn(0x7f9e04140ae0 msgr2=0x7f9e04140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 --2- 172.21.15.151:0/3277681661 >> v2:172.21.15.151:3300/0 conn(0x7f9e04140ae0 0x7f9e04140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 -- 172.21.15.151:0/3277681661 >> v1:172.21.15.172:6789/0 conn(0x7f9e0406b350 legacy=0x7f9e0406d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 -- 172.21.15.151:0/3277681661 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0406b8c0 con 0x7f9e040b3de0 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0a215700 1 --2- 172.21.15.151:0/3277681661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009f20 tx=0x7f9e00009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217bc07501422677 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e09213700 1 -- 172.21.15.151:0/3277681661 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0000e070 con 0x7f9e040b3de0 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e09213700 1 -- 172.21.15.151:0/3277681661 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0000b9d0 con 0x7f9e040b3de0 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e09213700 1 -- 172.21.15.151:0/3277681661 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e000092e0 con 0x7f9e040b3de0 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- 172.21.15.151:0/3277681661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 msgr2=0x7f9e0406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 --2- 172.21.15.151:0/3277681661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009f20 tx=0x7f9e00009840 comp rx=0 tx=0).stop 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- 172.21.15.151:0/3277681661 shutdown_connections 2023-09-30T01:57:54.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 --2- 172.21.15.151:0/3277681661 >> v2:172.21.15.151:3300/0 conn(0x7f9e04140ae0 0x7f9e04140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 --2- 172.21.15.151:0/3277681661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e0406d200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- 172.21.15.151:0/3277681661 >> 172.21.15.151:0/3277681661 conn(0x7f9e04070510 msgr2=0x7f9e04070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- 172.21.15.151:0/3277681661 shutdown_connections 2023-09-30T01:57:54.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.308+0000 7f9e0b217700 1 -- 172.21.15.151:0/3277681661 wait complete. 2023-09-30T01:57:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 Processor -- start 2023-09-30T01:57:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- start start 2023-09-30T01:57:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 0x7f9e040ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e040a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040ac4a0 con 0x7f9e04140ae0 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040ac930 con 0x7f9e0406b350 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040ad540 con 0x7f9e040b3de0 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55824/0 (socket says 172.21.15.151:55824) 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 -- 172.21.15.151:0/527938902 learned_addr learned my addr 172.21.15.151:0/527938902 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e09a14700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e040a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0a215700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 0x7f9e040ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 -- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 msgr2=0x7f9e040a8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e040a8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 -- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 msgr2=0x7f9e040ab6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 0x7f9e040ab6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 -- 172.21.15.151:0/527938902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00009ae0 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0a215700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 0x7f9e040ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e09a14700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e040a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0aa16700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00e8d0 tx=0x7f9dfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc00ee70 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc00c850 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e040ad880 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc00cc50 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e040abe60 con 0x7f9e04140ae0 2023-09-30T01:57:54.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9dfc013680 con 0x7f9e04140ae0 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.312+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e040a9600 con 0x7f9e04140ae0 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.316+0000 7f9dfa7fc700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 0x7f9de8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.316+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9dfc07c750 con 0x7f9e04140ae0 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.316+0000 7f9e0a215700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 0x7f9de8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.316+0000 7f9e0a215700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 0x7f9de8062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009880 tx=0x7f9e00018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.316+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9dfc04cf30 con 0x7f9e04140ae0 2023-09-30T01:57:54.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.464+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e040670a0 con 0x7f9de8060550 2023-09-30T01:57:54.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.468+0000 7f9dfa7fc700 1 -- 172.21.15.151:0/527938902 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+45866 (secure 0 0 0) 0x7f9e040670a0 con 0x7f9de8060550 2023-09-30T01:57:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 msgr2=0x7f9de8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 0x7f9de8062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009880 tx=0x7f9e00018000 comp rx=0 tx=0).stop 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 msgr2=0x7f9e040a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00e8d0 tx=0x7f9dfc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 shutdown_connections 2023-09-30T01:57:54.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9de8060550 0x7f9de8062a00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e04140ae0 0x7f9e040a9030 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e040b3de0 0x7f9e040a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 --2- 172.21.15.151:0/527938902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e0406b350 0x7f9e040ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 >> 172.21.15.151:0/527938902 conn(0x7f9e04070510 msgr2=0x7f9e0413f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 shutdown_connections 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.472+0000 7f9e0b217700 1 -- 172.21.15.151:0/527938902 wait complete. 2023-09-30T01:57:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T01:57:54.487 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-09-30T01:57:53.435688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42976,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931744,"statfs":{"total":773094113280,"available":773050105856,"internally_reserved":0,"allocated":2392064,"data_stored":1474082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.880316"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.544219+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:50.544219+0000","last_peered":"2023-09-30T01:57:50.544219+0000","last_clean":"2023-09-30T01:57:50.544219+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:50.544219+0000","last_undegraded":"2023-09-30T01:57:50.544219+0000","last_fullsized":"2023-09-30T01:57:50.544219+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:18:26.843921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.109930+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.109930+0000","last_peered":"2023-09-30T01:57:51.109930+0000","last_clean":"2023-09-30T01:57:51.109930+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.109930+0000","last_undegraded":"2023-09-30T01:57:51.109930+0000","last_fullsized":"2023-09-30T01:57:51.109930+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111284+0000","last_change":"2023-09-30T01:57:51.106990+0000","last_active":"2023-09-30T01:57:51.111284+0000","last_peered":"2023-09-30T01:57:51.111284+0000","last_clean":"2023-09-30T01:57:51.111284+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T01:57:51.111284+0000","last_undegraded":"2023-09-30T01:57:51.111284+0000","last_fullsized":"2023-09-30T01:57:51.111284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:40.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.547452+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.547452+0000","last_peered":"2023-09-30T01:57:50.547452+0000","last_clean":"2023-09-30T01:57:50.547452+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.547452+0000","last_undegraded":"2023-09-30T01:57:50.547452+0000","last_fullsized":"2023-09-30T01:57:50.547452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.701554+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.701554+0000","last_peered":"2023-09-30T01:57:50.701554+0000","last_clean":"2023-09-30T01:57:50.701554+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.701554+0000","last_undegraded":"2023-09-30T01:57:50.701554+0000","last_fullsized":"2023-09-30T01:57:50.701554+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.547375+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.547375+0000","last_peered":"2023-09-30T01:57:50.547375+0000","last_clean":"2023-09-30T01:57:50.547375+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.547375+0000","last_undegraded":"2023-09-30T01:57:50.547375+0000","last_fullsized":"2023-09-30T01:57:50.547375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549603+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.549603+0000","last_peered":"2023-09-30T01:57:50.549603+0000","last_clean":"2023-09-30T01:57:50.549603+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.549603+0000","last_undegraded":"2023-09-30T01:57:50.549603+0000","last_fullsized":"2023-09-30T01:57:50.549603+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.126241+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.126241+0000","last_peered":"2023-09-30T01:57:51.126241+0000","last_clean":"2023-09-30T01:57:51.126241+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.126241+0000","last_undegraded":"2023-09-30T01:57:51.126241+0000","last_fullsized":"2023-09-30T01:57:51.126241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365712,"statfs":{"total":96636764160,"available":96630489088,"internally_reserved":0,"allocated":696320,"data_stored":580579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2829999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5156,"kb_used_data":156,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366684,"statfs":{"total":96636764160,"available":96631484416,"internally_reserved":0,"allocated":159744,"data_stored":50316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5164,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366676,"statfs":{"total":96636764160,"available":96631476224,"internally_reserved":0,"allocated":167936,"data_stored":52599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365712,"statfs":{"total":96636764160,"available":96630489088,"internally_reserved":0,"allocated":696320,"data_stored":580192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0190000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:57:54.488 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:57:54.489 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T01:57:54.489 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T01:57:54.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:54.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 Processor -- start 2023-09-30T01:57:54.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 -- start start 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f17ac0ab460 0x7f17ac0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f17ac06b550 0x7f17ac13e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac06c8e0 con 0x7f17ac06b180 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac141ce0 con 0x7f17ac06b550 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2851700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac0abe70 con 0x7f17ac0ab460 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b184f700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f17ac06b180 0x7f17ac1415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43380/0 (socket says 172.21.15.151:43380) 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b184f700 1 -- 172.21.15.151:0/572799201 learned_addr learned my addr 172.21.15.151:0/572799201 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:54.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 --2- 172.21.15.151:0/572799201 >> v2:172.21.15.151:3300/0 conn(0x7f17ac06b550 0x7f17ac13e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b104e700 1 --2- 172.21.15.151:0/572799201 >> v2:172.21.15.172:3300/0 conn(0x7f17ac0ab460 0x7f17ac0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b084d700 1 -- 172.21.15.151:0/572799201 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635695951 0 0) 0x7f17ac06c8e0 con 0x7f17ac06b180 2023-09-30T01:57:54.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b084d700 1 -- 172.21.15.151:0/572799201 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003570 con 0x7f17ac06b180 2023-09-30T01:57:54.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 -- 172.21.15.151:0/572799201 >> v2:172.21.15.172:3300/0 conn(0x7f17ac0ab460 msgr2=0x7f17ac0ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:54.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 --2- 172.21.15.151:0/572799201 >> v2:172.21.15.172:3300/0 conn(0x7f17ac0ab460 0x7f17ac0ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 -- 172.21.15.151:0/572799201 >> v1:172.21.15.151:6789/0 conn(0x7f17ac06b180 legacy=0x7f17ac1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 -- 172.21.15.151:0/572799201 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac0a9b00 con 0x7f17ac06b550 2023-09-30T01:57:54.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b2050700 1 --2- 172.21.15.151:0/572799201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b550 0x7f17ac13e070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17ac13f150 tx=0x7f17a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59847b92abc13b5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b084d700 1 -- 172.21.15.151:0/572799201 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a400e070 con 0x7f17ac06b550 2023-09-30T01:57:54.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b084d700 1 -- 172.21.15.151:0/572799201 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a400bab0 con 0x7f17ac06b550 2023-09-30T01:57:54.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.580+0000 7f17b084d700 1 -- 172.21.15.151:0/572799201 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a400be50 con 0x7f17ac06b550 2023-09-30T01:57:54.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/572799201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b550 msgr2=0x7f17ac13e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- 172.21.15.151:0/572799201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b550 0x7f17ac13e070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17ac13f150 tx=0x7f17a4009780 comp rx=0 tx=0).stop 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/572799201 shutdown_connections 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- 172.21.15.151:0/572799201 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b550 0x7f17ac13e070 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- 172.21.15.151:0/572799201 >> v2:172.21.15.172:3300/0 conn(0x7f17ac0ab460 0x7f17ac0ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/572799201 >> 172.21.15.151:0/572799201 conn(0x7f17ac070510 msgr2=0x7f17ac070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/572799201 shutdown_connections 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/572799201 wait complete. 2023-09-30T01:57:54.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 Processor -- start 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- start start 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 0x7f17ac145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 0x7f17ac153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17ac0ab460 0x7f17ac1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac143010 con 0x7f17ac06b180 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac142ba0 con 0x7f17ac0ab460 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac142d60 con 0x7f17ac06b550 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b104e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 0x7f17ac153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b104e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 0x7f17ac153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44802/0 (socket says 172.21.15.151:44802) 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b104e700 1 -- 172.21.15.151:0/2676168547 learned_addr learned my addr 172.21.15.151:0/2676168547 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 0x7f17ac145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2050700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17ac0ab460 0x7f17ac1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 -- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 msgr2=0x7f17ac153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 0x7f17ac153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 -- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17ac0ab460 msgr2=0x7f17ac1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17ac0ab460 0x7f17ac1434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 -- 172.21.15.151:0/2676168547 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4009a20 con 0x7f17ac06b180 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b184f700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 0x7f17ac145620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17a80058d0 tx=0x7f17a8007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a8005b30 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a8005c90 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a8007320 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac154ae0 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.584+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac143c00 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac154d00 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f17a801b3b0 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17a1ffb700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 0x7f1790062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f17a807b930 con 0x7f17ac06b180 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17b104e700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 0x7f1790062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.588+0000 7f17b104e700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 0x7f1790062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17ac144e50 tx=0x7f179c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.592+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f17a804c110 con 0x7f17ac06b180 2023-09-30T01:57:54.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17ac0008d0 con 0x7f17ac06b180 2023-09-30T01:57:54.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17a1ffb700 1 -- 172.21.15.151:0/2676168547 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f17a804f730 con 0x7f17ac06b180 2023-09-30T01:57:54.777 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:54.777 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 msgr2=0x7f1790062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 0x7f1790062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17ac144e50 tx=0x7f179c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 msgr2=0x7f17ac145620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 0x7f17ac145620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17a80058d0 tx=0x7f17a8007d10 comp rx=0 tx=0).stop 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 shutdown_connections 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1790060550 0x7f1790062a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17ac0ab460 0x7f17ac1434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17ac06b550 0x7f17ac153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 --2- 172.21.15.151:0/2676168547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f17ac06b180 0x7f17ac145620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 >> 172.21.15.151:0/2676168547 conn(0x7f17ac070510 msgr2=0x7f17ac0aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 shutdown_connections 2023-09-30T01:57:54.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:54.772+0000 7f17b2851700 1 -- 172.21.15.151:0/2676168547 wait complete. 2023-09-30T01:57:57.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:57.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 Processor -- start 2023-09-30T01:57:57.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 -- start start 2023-09-30T01:57:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25f413de70 0x7f25f4148e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f40aa210 con 0x7f25f40b0a20 2023-09-30T01:57:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f40a8fb0 con 0x7f25f4141010 2023-09-30T01:57:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fb18d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f40b0df0 con 0x7f25f413de70 2023-09-30T01:57:57.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa18b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f25f40b0a20 0x7f25f4140900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43390/0 (socket says 172.21.15.151:43390) 2023-09-30T01:57:57.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa18b700 1 -- 172.21.15.151:0/3915773187 learned_addr learned my addr 172.21.15.151:0/3915773187 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:57:57.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa98c700 1 --2- 172.21.15.151:0/3915773187 >> v2:172.21.15.172:3300/0 conn(0x7f25f413de70 0x7f25f4148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192645416 0 0) 0x7f25f40aa210 con 0x7f25f40b0a20 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0003570 con 0x7f25f40b0a20 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408325098 0 0) 0x7f25e0003570 con 0x7f25f40b0a20 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa98c700 1 -- 172.21.15.151:0/3915773187 >> v1:172.21.15.172:6789/0 conn(0x7f25f4141010 legacy=0x7f25f413d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa98c700 1 -- 172.21.15.151:0/3915773187 >> v1:172.21.15.151:6789/0 conn(0x7f25f40b0a20 legacy=0x7f25f4140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa98c700 1 -- 172.21.15.151:0/3915773187 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f40a9ef0 con 0x7f25f413de70 2023-09-30T01:57:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.900+0000 7f25fa98c700 1 --2- 172.21.15.151:0/3915773187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f413de70 0x7f25f4148e60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0098e0 tx=0x7f25ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=297d265a88ce1384 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:57.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e4001470 con 0x7f25f413de70 2023-09-30T01:57:57.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec011420 con 0x7f25f413de70 2023-09-30T01:57:57.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25f9189700 1 -- 172.21.15.151:0/3915773187 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0117a0 con 0x7f25f413de70 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/3915773187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f413de70 msgr2=0x7f25f4148e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 --2- 172.21.15.151:0/3915773187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f413de70 0x7f25f4148e60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0098e0 tx=0x7f25ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/3915773187 shutdown_connections 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 --2- 172.21.15.151:0/3915773187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f413de70 0x7f25f4148e60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/3915773187 >> 172.21.15.151:0/3915773187 conn(0x7f25f4070510 msgr2=0x7f25f4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:57.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/3915773187 shutdown_connections 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/3915773187 wait complete. 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 Processor -- start 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- start start 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f40b0a20 0x7f25f4118d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25f413de70 0x7f25f413b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f411b1d0 con 0x7f25f413de70 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f411a040 con 0x7f25f40b0a20 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f413bd90 con 0x7f25f4141010 2023-09-30T01:57:57.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44816/0 (socket says 172.21.15.151:44816) 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 -- 172.21.15.151:0/2302478630 learned_addr learned my addr 172.21.15.151:0/2302478630 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25f998a700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25f413de70 0x7f25f413b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa18b700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f40b0a20 0x7f25f4118d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 -- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f40b0a20 msgr2=0x7f25f4118d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:57.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f40b0a20 0x7f25f4118d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 -- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25f413de70 msgr2=0x7f25f413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25f413de70 0x7f25f413b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 -- 172.21.15.151:0/2302478630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009420 con 0x7f25f4141010 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fa98c700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009f80 tx=0x7f25ec016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec01a660 con 0x7f25f4141010 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec01a7e0 con 0x7f25f4141010 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f4144440 con 0x7f25f4141010 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec01aba0 con 0x7f25f4141010 2023-09-30T01:57:57.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.904+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f4119b20 con 0x7f25f4141010 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.908+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f41425f0 con 0x7f25f4141010 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.908+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f25ec02d440 con 0x7f25f4141010 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.908+0000 7f25ea7fc700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 0x7f25d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.912+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f25ec07e4c0 con 0x7f25f4141010 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.912+0000 7f25fa18b700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 0x7f25d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:57.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.912+0000 7f25fa18b700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 0x7f25d8062a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25e4005b80 tx=0x7f25e4000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:57.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:57.916+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f25f41425f0 con 0x7f25f4141010 2023-09-30T01:57:58.115 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.108+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25f41425f0 con 0x7f25f4141010 2023-09-30T01:57:58.115 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25ea7fc700 1 -- 172.21.15.151:0/2302478630 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f25f41425f0 con 0x7f25f4141010 2023-09-30T01:57:58.116 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:57:58.116 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 msgr2=0x7f25d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 0x7f25d8062a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25e4005b80 tx=0x7f25e4000bc0 comp rx=0 tx=0).stop 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 msgr2=0x7f25f413b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009f80 tx=0x7f25ec016000 comp rx=0 tx=0).stop 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 shutdown_connections 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f25d8060550 0x7f25d8062a00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25f4141010 0x7f25f413b850 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25f413de70 0x7f25f413b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:58.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 --2- 172.21.15.151:0/2302478630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25f40b0a20 0x7f25f4118d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:58.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.112+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 >> 172.21.15.151:0/2302478630 conn(0x7f25f4070510 msgr2=0x7f25f40b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:58.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.116+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 shutdown_connections 2023-09-30T01:57:58.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:57:58.116+0000 7f25fb18d700 1 -- 172.21.15.151:0/2302478630 wait complete. 2023-09-30T01:58:01.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:01.244 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 Processor -- start 2023-09-30T01:58:01.244 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- start start 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1a640b4510 0x7f1a6406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1a6406b590 0x7f1a640ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a640aea80 0x7f1a640aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a640af3f0 con 0x7f1a640b4510 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6406b9d0 con 0x7f1a6406b590 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64141a50 con 0x7f1a640aea80 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1a640b4510 0x7f1a6406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1a640b4510 0x7f1a6406b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39826/0 (socket says 172.21.15.151:39826) 2023-09-30T01:58:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 -- 172.21.15.151:0/3775935875 learned_addr learned my addr 172.21.15.151:0/3775935875 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6992d700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a640aea80 0x7f1a640aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6892b700 1 --2- 172.21.15.151:0/3775935875 >> v2:172.21.15.172:3300/0 conn(0x7f1a6406b590 0x7f1a640ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 -- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a640aea80 msgr2=0x7f1a640aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a640aea80 0x7f1a640aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 -- 172.21.15.151:0/3775935875 >> v2:172.21.15.172:3300/0 conn(0x7f1a6406b590 msgr2=0x7f1a640ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 --2- 172.21.15.151:0/3775935875 >> v2:172.21.15.172:3300/0 conn(0x7f1a6406b590 0x7f1a640ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 -- 172.21.15.151:0/3775935875 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6406bf90 con 0x7f1a640b4510 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6912c700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6406b050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a64064530 tx=0x7f1a60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4779ff8a1708cbce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a5bfff700 1 -- 172.21.15.151:0/3775935875 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000b640 con 0x7f1a640b4510 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a5bfff700 1 -- 172.21.15.151:0/3775935875 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6000b7a0 con 0x7f1a640b4510 2023-09-30T01:58:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a5bfff700 1 -- 172.21.15.151:0/3775935875 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000bb40 con 0x7f1a640b4510 2023-09-30T01:58:01.248 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 msgr2=0x7f1a6406b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6406b050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a64064530 tx=0x7f1a60009250 comp rx=0 tx=0).stop 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- 172.21.15.151:0/3775935875 shutdown_connections 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a640aea80 0x7f1a640aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/3775935875 >> v2:172.21.15.172:3300/0 conn(0x7f1a6406b590 0x7f1a640ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/3775935875 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6406b050 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- 172.21.15.151:0/3775935875 >> 172.21.15.151:0/3775935875 conn(0x7f1a64070f90 msgr2=0x7f1a640b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- 172.21.15.151:0/3775935875 shutdown_connections 2023-09-30T01:58:01.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.240+0000 7f1a6a12e700 1 -- 172.21.15.151:0/3775935875 wait complete. 2023-09-30T01:58:01.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.244+0000 7f1a6a12e700 1 Processor -- start 2023-09-30T01:58:01.250 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.244+0000 7f1a6a12e700 1 -- start start 2023-09-30T01:58:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 0x7f1a640aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a640aea80 0x7f1a641441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6414c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640aaad0 con 0x7f1a640b4510 2023-09-30T01:58:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6414a450 con 0x7f1a640aea80 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640aaff0 con 0x7f1a6406b590 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6912c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 0x7f1a640aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6912c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 0x7f1a640aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:36848/0 (socket says 172.21.15.151:36848) 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6414c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6912c700 1 -- 172.21.15.151:0/751395606 learned_addr learned my addr 172.21.15.151:0/751395606 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6892b700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a640aea80 0x7f1a641441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 -- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 msgr2=0x7f1a640aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 0x7f1a640aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 -- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a640aea80 msgr2=0x7f1a641441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a640aea80 0x7f1a641441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 -- 172.21.15.151:0/751395606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a600094f0 con 0x7f1a640b4510 2023-09-30T01:58:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6992d700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6414c920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009e50 tx=0x7f1a5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c00caa0 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c00f040 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a640a9bf0 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c00ce50 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6414d080 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6414d300 con 0x7f1a640b4510 2023-09-30T01:58:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.248+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1a5c014690 con 0x7f1a640b4510 2023-09-30T01:58:01.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.252+0000 7f1a597fa700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 0x7f1a48062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:01.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.252+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f1a5c07d1e0 con 0x7f1a640b4510 2023-09-30T01:58:01.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.252+0000 7f1a6912c700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 0x7f1a48062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:01.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.252+0000 7f1a6912c700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 0x7f1a48062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1a60009290 tx=0x7f1a60017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:01.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.260+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a5c0513b0 con 0x7f1a640b4510 2023-09-30T01:58:01.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1a640670a0 con 0x7f1a640b4510 2023-09-30T01:58:01.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a597fa700 1 -- 172.21.15.151:0/751395606 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1a5c050f40 con 0x7f1a640b4510 2023-09-30T01:58:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:01.457 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 msgr2=0x7f1a48062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 0x7f1a48062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1a60009290 tx=0x7f1a60017000 comp rx=0 tx=0).stop 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 msgr2=0x7f1a6414c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6414c920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009e50 tx=0x7f1a5c00b040 comp rx=0 tx=0).stop 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 shutdown_connections 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1a48060550 0x7f1a48062a00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a640b4510 0x7f1a6414c920 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a640aea80 0x7f1a641441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 --2- 172.21.15.151:0/751395606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a6406b590 0x7f1a640aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:01.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 >> 172.21.15.151:0/751395606 conn(0x7f1a64070f90 msgr2=0x7f1a640b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:01.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 shutdown_connections 2023-09-30T01:58:01.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:01.452+0000 7f1a6a12e700 1 -- 172.21.15.151:0/751395606 wait complete. 2023-09-30T01:58:04.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:04.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 Processor -- start 2023-09-30T01:58:04.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 -- start start 2023-09-30T01:58:04.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f32000aea80 0x7f320006a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f320006df10 0x7f32000acc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320006eae0 con 0x7f320006da90 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320006a750 con 0x7f32000aea80 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f32083b4700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32000aee50 con 0x7f320006df10 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f320594f700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f320006da90 0x7f32000ac550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38896/0 (socket says 172.21.15.151:38896) 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f320594f700 1 -- 172.21.15.151:0/243017509 learned_addr learned my addr 172.21.15.151:0/243017509 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:04.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f3206150700 1 --2- 172.21.15.151:0/243017509 >> v2:172.21.15.151:3300/0 conn(0x7f32000aea80 0x7f320006a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f3206951700 1 --2- 172.21.15.151:0/243017509 >> v2:172.21.15.172:3300/0 conn(0x7f320006df10 0x7f32000acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f320514e700 1 -- 172.21.15.151:0/243017509 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847380590 0 0) 0x7f320006eae0 con 0x7f320006da90 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.580+0000 7f320514e700 1 -- 172.21.15.151:0/243017509 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec003570 con 0x7f320006da90 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f3206150700 1 -- 172.21.15.151:0/243017509 >> v2:172.21.15.172:3300/0 conn(0x7f320006df10 msgr2=0x7f32000acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f3206150700 1 --2- 172.21.15.151:0/243017509 >> v2:172.21.15.172:3300/0 conn(0x7f320006df10 0x7f32000acc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f3206150700 1 -- 172.21.15.151:0/243017509 >> v1:172.21.15.151:6789/0 conn(0x7f320006da90 legacy=0x7f32000ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f3206150700 1 -- 172.21.15.151:0/243017509 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32000ac1b0 con 0x7f32000aea80 2023-09-30T01:58:04.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f3206150700 1 --2- 172.21.15.151:0/243017509 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320006a210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32000ad920 tx=0x7f31fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc84d997b9e57d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:04.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f320514e700 1 -- 172.21.15.151:0/243017509 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc011630 con 0x7f32000aea80 2023-09-30T01:58:04.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f320514e700 1 -- 172.21.15.151:0/243017509 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc011790 con 0x7f32000aea80 2023-09-30T01:58:04.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f320514e700 1 -- 172.21.15.151:0/243017509 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc011b50 con 0x7f32000aea80 2023-09-30T01:58:04.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 -- 172.21.15.151:0/243017509 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 msgr2=0x7f320006a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 --2- 172.21.15.151:0/243017509 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320006a210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32000ad920 tx=0x7f31fc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 -- 172.21.15.151:0/243017509 shutdown_connections 2023-09-30T01:58:04.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 --2- 172.21.15.151:0/243017509 >> v2:172.21.15.172:3300/0 conn(0x7f320006df10 0x7f32000acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 --2- 172.21.15.151:0/243017509 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320006a210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 -- 172.21.15.151:0/243017509 >> 172.21.15.151:0/243017509 conn(0x7f3200070510 msgr2=0x7f3200070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:04.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 -- 172.21.15.151:0/243017509 shutdown_connections 2023-09-30T01:58:04.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.584+0000 7f32083b4700 1 -- 172.21.15.151:0/243017509 wait complete. 2023-09-30T01:58:04.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 Processor -- start 2023-09-30T01:58:04.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 -- start start 2023-09-30T01:58:04.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320006da90 0x7f3200119a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 0x7f32000ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32000abf00 con 0x7f32000aea80 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320014ca00 con 0x7f320006da90 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32000a90b0 con 0x7f320006df10 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f3206951700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 0x7f32000ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f3206951700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39852/0 (socket says 172.21.15.151:39852) 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f3206951700 1 -- 172.21.15.151:0/926368760 learned_addr learned my addr 172.21.15.151:0/926368760 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f3206150700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320006da90 0x7f3200119a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 -- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320006da90 msgr2=0x7f3200119a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320006da90 0x7f3200119a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 -- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 msgr2=0x7f320014ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 -- 172.21.15.151:0/926368760 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc009420 con 0x7f320006df10 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f3206951700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f320594f700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 0x7f32000ab9c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31f000c5b0 tx=0x7f31f0007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f0005b30 con 0x7f320006df10 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f0005cb0 con 0x7f320006df10 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320011a140 con 0x7f320006df10 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.588+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f0007320 con 0x7f320006df10 2023-09-30T01:58:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.592+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320014d570 con 0x7f320006df10 2023-09-30T01:58:04.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.592+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32000670a0 con 0x7f320006df10 2023-09-30T01:58:04.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.592+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f31f001b3b0 con 0x7f320006df10 2023-09-30T01:58:04.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.592+0000 7f31f67fc700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 0x7f31e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.592+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f31f007b9d0 con 0x7f320006df10 2023-09-30T01:58:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.596+0000 7f3206150700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 0x7f31e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:04.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.596+0000 7f3206150700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 0x7f31e4062a00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0093f0 tx=0x7f31fc017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:04.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.604+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f31f004c1b0 con 0x7f320006df10 2023-09-30T01:58:04.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.800+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f320014cb40 con 0x7f320006df10 2023-09-30T01:58:04.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.800+0000 7f31f67fc700 1 -- 172.21.15.151:0/926368760 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31f004f7d0 con 0x7f320006df10 2023-09-30T01:58:04.806 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:04.806 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:04.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 msgr2=0x7f31e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 0x7f31e4062a00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0093f0 tx=0x7f31fc017000 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 msgr2=0x7f32000ab9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 0x7f32000ab9c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31f000c5b0 tx=0x7f31f0007d50 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 shutdown_connections 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f31e4060550 0x7f31e4062a00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32000aea80 0x7f320014ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f320006df10 0x7f32000ab9c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 --2- 172.21.15.151:0/926368760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320006da90 0x7f3200119a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 >> 172.21.15.151:0/926368760 conn(0x7f3200070510 msgr2=0x7f3200140e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:04.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 shutdown_connections 2023-09-30T01:58:04.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:04.804+0000 7f32083b4700 1 -- 172.21.15.151:0/926368760 wait complete. 2023-09-30T01:58:07.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:07.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 Processor -- start 2023-09-30T01:58:07.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 -- start start 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f09f80acf40 0x7f09f8137490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f806c440 con 0x7f09f8069f80 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f806a350 con 0x7f09f80acf40 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.929+0000 7f09fd1a1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f806c690 con 0x7f09f806df80 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:36874/0 (socket says 172.21.15.151:36874) 2023-09-30T01:58:07.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 -- 172.21.15.151:0/3372630960 learned_addr learned my addr 172.21.15.151:0/3372630960 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09f77fe700 1 --2- 172.21.15.151:0/3372630960 >> v2:172.21.15.172:3300/0 conn(0x7f09f80acf40 0x7f09f8137490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 -- 172.21.15.151:0/3372630960 >> v2:172.21.15.172:3300/0 conn(0x7f09f80acf40 msgr2=0x7f09f8137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 --2- 172.21.15.151:0/3372630960 >> v2:172.21.15.172:3300/0 conn(0x7f09f80acf40 0x7f09f8137490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 -- 172.21.15.151:0/3372630960 >> v1:172.21.15.172:6789/0 conn(0x7f09f8069f80 legacy=0x7f09f8136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 -- 172.21.15.151:0/3372630960 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8137a30 con 0x7f09f806df80 2023-09-30T01:58:07.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fc9a0700 1 --2- 172.21.15.151:0/3372630960 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009f20 tx=0x7f09f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6a417f943b769c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09f6ffd700 1 -- 172.21.15.151:0/3372630960 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f00092e0 con 0x7f09f806df80 2023-09-30T01:58:07.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09f6ffd700 1 -- 172.21.15.151:0/3372630960 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000ba80 con 0x7f09f806df80 2023-09-30T01:58:07.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09f6ffd700 1 -- 172.21.15.151:0/3372630960 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f000be40 con 0x7f09f806df80 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- 172.21.15.151:0/3372630960 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 msgr2=0x7f09f806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 --2- 172.21.15.151:0/3372630960 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009f20 tx=0x7f09f0009840 comp rx=0 tx=0).stop 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- 172.21.15.151:0/3372630960 shutdown_connections 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 --2- 172.21.15.151:0/3372630960 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f806df80 0x7f09f806e420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 --2- 172.21.15.151:0/3372630960 >> v2:172.21.15.172:3300/0 conn(0x7f09f80acf40 0x7f09f8137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- 172.21.15.151:0/3372630960 >> 172.21.15.151:0/3372630960 conn(0x7f09f80b08c0 msgr2=0x7f09f80b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- 172.21.15.151:0/3372630960 shutdown_connections 2023-09-30T01:58:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- 172.21.15.151:0/3372630960 wait complete. 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 Processor -- start 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.933+0000 7f09fd1a1700 1 -- start start 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f806df80 0x7f09f8139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f80acf40 0x7f09f8135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8139a60 con 0x7f09f8069f80 2023-09-30T01:58:07.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8139740 con 0x7f09f806df80 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f81386a0 con 0x7f09f80acf40 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39862/0 (socket says 172.21.15.151:39862) 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 -- 172.21.15.151:0/184810437 learned_addr learned my addr 172.21.15.151:0/184810437 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f77fe700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f806df80 0x7f09f8139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fc9a0700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f80acf40 0x7f09f8135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 -- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f80acf40 msgr2=0x7f09f8135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f80acf40 0x7f09f8135ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 -- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f806df80 msgr2=0x7f09f8139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f806df80 0x7f09f8139200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 -- 172.21.15.151:0/184810437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f0009ae0 con 0x7f09f8069f80 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09f7fff700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c590 tx=0x7f09e800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8011590 con 0x7f09f8069f80 2023-09-30T01:58:07.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 -- 172.21.15.151:0/184810437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8138920 con 0x7f09f8069f80 2023-09-30T01:58:07.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8011710 con 0x7f09f8069f80 2023-09-30T01:58:07.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8011ad0 con 0x7f09f8069f80 2023-09-30T01:58:07.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09fd1a1700 1 -- 172.21.15.151:0/184810437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f81361b0 con 0x7f09f8069f80 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.937+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d00050b0 con 0x7f09f8069f80 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.941+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f09e801b3b0 con 0x7f09f8069f80 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.941+0000 7f09dffff700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 0x7f09e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.941+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f09e807ba70 con 0x7f09f8069f80 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.941+0000 7f09f77fe700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 0x7f09e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.941+0000 7f09f77fe700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 0x7f09e4062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f813a670 tx=0x7f09ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:07.949+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f09e804c250 con 0x7f09f8069f80 2023-09-30T01:58:08.149 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.145+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09d0005360 con 0x7f09f8069f80 2023-09-30T01:58:08.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.145+0000 7f09dffff700 1 -- 172.21.15.151:0/184810437 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f09e8017080 con 0x7f09f8069f80 2023-09-30T01:58:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:08.150 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 msgr2=0x7f09e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 0x7f09e4062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09f813a670 tx=0x7f09ec008040 comp rx=0 tx=0).stop 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 msgr2=0x7f09f8140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c590 tx=0x7f09e800c8a0 comp rx=0 tx=0).stop 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 shutdown_connections 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f09e4060550 0x7f09e4062a00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09f80acf40 0x7f09f8135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09f806df80 0x7f09f8139200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:08.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 --2- 172.21.15.151:0/184810437 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09f8069f80 0x7f09f8140890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:08.153 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 >> 172.21.15.151:0/184810437 conn(0x7f09f80b08c0 msgr2=0x7f09f80ad310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:08.153 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 shutdown_connections 2023-09-30T01:58:08.153 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:08.149+0000 7f09ddffb700 1 -- 172.21.15.151:0/184810437 wait complete. 2023-09-30T01:58:11.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:11.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 Processor -- start 2023-09-30T01:58:11.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 -- start start 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ae00b3f60 0x7f8ae006d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ae0140ac0 0x7f8ae0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae006c720 con 0x7f8ae00b3b90 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae006ca00 con 0x7f8ae00b3f60 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.273+0000 7f8adf59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae006b810 con 0x7f8ae0140ac0 2023-09-30T01:58:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ae00b3f60 0x7f8ae006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ae00b3f60 0x7f8ae006d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49258/0 (socket says 172.21.15.151:49258) 2023-09-30T01:58:11.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 -- 172.21.15.151:0/1400666834 learned_addr learned my addr 172.21.15.151:0/1400666834 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:11.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8aded9d700 1 --2- 172.21.15.151:0/1400666834 >> v2:172.21.15.172:3300/0 conn(0x7f8ae0140ac0 0x7f8ae0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 -- 172.21.15.151:0/1400666834 >> v2:172.21.15.172:3300/0 conn(0x7f8ae0140ac0 msgr2=0x7f8ae0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 --2- 172.21.15.151:0/1400666834 >> v2:172.21.15.172:3300/0 conn(0x7f8ae0140ac0 0x7f8ae0140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 -- 172.21.15.151:0/1400666834 >> v1:172.21.15.172:6789/0 conn(0x7f8ae00b3b90 legacy=0x7f8ae006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8addd9b700 1 -- 172.21.15.151:0/1400666834 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303111132 0 0) 0x7f8ae006c720 con 0x7f8ae00b3b90 2023-09-30T01:58:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 -- 172.21.15.151:0/1400666834 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00b0170 con 0x7f8ae00b3f60 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8ad5d9b700 1 --2- 172.21.15.151:0/1400666834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8ae006c560 tx=0x7f8ac8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc224e63f00aa313 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8addd9b700 1 -- 172.21.15.151:0/1400666834 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8009e70 con 0x7f8ae00b3f60 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8addd9b700 1 -- 172.21.15.151:0/1400666834 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac800b860 con 0x7f8ae00b3f60 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8addd9b700 1 -- 172.21.15.151:0/1400666834 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac800bc20 con 0x7f8ae00b3f60 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- 172.21.15.151:0/1400666834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 msgr2=0x7f8ae006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 --2- 172.21.15.151:0/1400666834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8ae006c560 tx=0x7f8ac8009250 comp rx=0 tx=0).stop 2023-09-30T01:58:11.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- 172.21.15.151:0/1400666834 shutdown_connections 2023-09-30T01:58:11.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 --2- 172.21.15.151:0/1400666834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae006d810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 --2- 172.21.15.151:0/1400666834 >> v2:172.21.15.172:3300/0 conn(0x7f8ae0140ac0 0x7f8ae0140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- 172.21.15.151:0/1400666834 >> 172.21.15.151:0/1400666834 conn(0x7f8ae0063990 msgr2=0x7f8ae0070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:11.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- 172.21.15.151:0/1400666834 shutdown_connections 2023-09-30T01:58:11.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- 172.21.15.151:0/1400666834 wait complete. 2023-09-30T01:58:11.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 Processor -- start 2023-09-30T01:58:11.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 -- start start 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ae00b3b90 0x7f8ae011dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.277+0000 7f8adf59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae011cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae011d8d0 con 0x7f8ae00b3f60 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae011f290 con 0x7f8ae00b3b90 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00acf20 con 0x7f8ae0140ac0 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8ad5d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae011cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58652/0 (socket says 172.21.15.151:58652) 2023-09-30T01:58:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 -- 172.21.15.151:0/2532834964 learned_addr learned my addr 172.21.15.151:0/2532834964 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8ade59c700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ae00b3b90 0x7f8ae011dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 -- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ae00b3b90 msgr2=0x7f8ae011dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ae00b3b90 0x7f8ae011dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 -- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 msgr2=0x7f8ae011cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae011cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 -- 172.21.15.151:0/2532834964 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80094f0 con 0x7f8ae0140ac0 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8aded9d700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009ec0 tx=0x7f8ad800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:11.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800cb70 con 0x7f8ae0140ac0 2023-09-30T01:58:11.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad800f040 con 0x7f8ae0140ac0 2023-09-30T01:58:11.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae011d480 con 0x7f8ae0140ac0 2023-09-30T01:58:11.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8014480 con 0x7f8ae0140ac0 2023-09-30T01:58:11.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae014cdf0 con 0x7f8ae0140ac0 2023-09-30T01:58:11.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.281+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae011da40 con 0x7f8ae0140ac0 2023-09-30T01:58:11.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.285+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8ad8014830 con 0x7f8ae0140ac0 2023-09-30T01:58:11.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.285+0000 7f8ad6ffd700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 0x7f8ac4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:11.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.285+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8ad807ce70 con 0x7f8ae0140ac0 2023-09-30T01:58:11.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.285+0000 7f8ade59c700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 0x7f8ac4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:11.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.285+0000 7f8ade59c700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 0x7f8ac4062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0003960 tx=0x7f8ad0007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:11.295 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.293+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8ad804d650 con 0x7f8ae0140ac0 2023-09-30T01:58:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.481+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8ae00008d0 con 0x7f8ae0140ac0 2023-09-30T01:58:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.485+0000 7f8ad6ffd700 1 -- 172.21.15.151:0/2532834964 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8ad8018100 con 0x7f8ae0140ac0 2023-09-30T01:58:11.488 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:11.488 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:11.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 msgr2=0x7f8ac4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 0x7f8ac4062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0003960 tx=0x7f8ad0007ea0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 msgr2=0x7f8ae014c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009ec0 tx=0x7f8ad800b040 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 shutdown_connections 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ac4060550 0x7f8ac4062a00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ae0140ac0 0x7f8ae014c710 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ae00b3f60 0x7f8ae011cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 --2- 172.21.15.151:0/2532834964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ae00b3b90 0x7f8ae011dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 >> 172.21.15.151:0/2532834964 conn(0x7f8ae0063990 msgr2=0x7f8ae01402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:11.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 shutdown_connections 2023-09-30T01:58:11.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:11.489+0000 7f8adf59e700 1 -- 172.21.15.151:0/2532834964 wait complete. 2023-09-30T01:58:14.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:14.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 Processor -- start 2023-09-30T01:58:14.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 -- start start 2023-09-30T01:58:14.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f86140b0960 0x7f861406d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8614140c10 0x7f8614141050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86140b0d30 con 0x7f861406b3d0 2023-09-30T01:58:14.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86140ae490 con 0x7f86140b0960 2023-09-30T01:58:14.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861c43a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861406c690 con 0x7f8614140c10 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f86140b0960 0x7f861406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f86199d5700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f861406b3d0 0x7f861406d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:46974/0 (socket says 172.21.15.151:46974) 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f86199d5700 1 -- 172.21.15.151:0/156270235 learned_addr learned my addr 172.21.15.151:0/156270235 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a9d7700 1 --2- 172.21.15.151:0/156270235 >> v2:172.21.15.172:3300/0 conn(0x7f8614140c10 0x7f8614141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 -- 172.21.15.151:0/156270235 >> v2:172.21.15.172:3300/0 conn(0x7f8614140c10 msgr2=0x7f8614141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 --2- 172.21.15.151:0/156270235 >> v2:172.21.15.172:3300/0 conn(0x7f8614140c10 0x7f8614141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 -- 172.21.15.151:0/156270235 >> v1:172.21.15.151:6789/0 conn(0x7f861406b3d0 legacy=0x7f861406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 -- 172.21.15.151:0/156270235 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861406c970 con 0x7f86140b0960 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.613+0000 7f861a1d6700 1 --2- 172.21.15.151:0/156270235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86140b0960 0x7f861406d270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f861406bda0 tx=0x7f86100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cb7708ccbbc199 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f86191d4700 1 -- 172.21.15.151:0/156270235 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610011630 con 0x7f86140b0960 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f86191d4700 1 -- 172.21.15.151:0/156270235 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8610011790 con 0x7f86140b0960 2023-09-30T01:58:14.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f86191d4700 1 -- 172.21.15.151:0/156270235 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610011b30 con 0x7f86140b0960 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- 172.21.15.151:0/156270235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86140b0960 msgr2=0x7f861406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- 172.21.15.151:0/156270235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86140b0960 0x7f861406d270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f861406bda0 tx=0x7f86100091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- 172.21.15.151:0/156270235 shutdown_connections 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- 172.21.15.151:0/156270235 >> v2:172.21.15.172:3300/0 conn(0x7f8614140c10 0x7f8614141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- 172.21.15.151:0/156270235 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86140b0960 0x7f861406d270 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- 172.21.15.151:0/156270235 >> 172.21.15.151:0/156270235 conn(0x7f8614070510 msgr2=0x7f8614070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- 172.21.15.151:0/156270235 shutdown_connections 2023-09-30T01:58:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- 172.21.15.151:0/156270235 wait complete. 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 Processor -- start 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- start start 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86140b0960 0x7f861413bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8614140c10 0x7f861413c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86140ac1b0 con 0x7f861406b3d0 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86140ad460 con 0x7f86140b0960 2023-09-30T01:58:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861c43a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86140aa1c0 con 0x7f8614140c10 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861a1d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861a1d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49284/0 (socket says 172.21.15.151:49284) 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861a1d6700 1 -- 172.21.15.151:0/3196834253 learned_addr learned my addr 172.21.15.151:0/3196834253 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f861a9d7700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8614140c10 0x7f861413c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.617+0000 7f86199d5700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86140b0960 0x7f861413bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 -- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8614140c10 msgr2=0x7f861413c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8614140c10 0x7f861413c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 -- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86140b0960 msgr2=0x7f861413bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86140b0960 0x7f861413bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 -- 172.21.15.151:0/3196834253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8610009420 con 0x7f861406b3d0 2023-09-30T01:58:14.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861a1d6700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f86140b1150 tx=0x7f8610017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610000d10 con 0x7f861406b3d0 2023-09-30T01:58:14.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8610011b10 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861413c870 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610025640 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86140ada20 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8610025a30 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86140670a0 con 0x7f861406b3d0 2023-09-30T01:58:14.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 0x7f85f8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f861007dd00 con 0x7f861406b3d0 2023-09-30T01:58:14.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.621+0000 7f86199d5700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 0x7f85f8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.625+0000 7f86199d5700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 0x7f85f8062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8614144f10 tx=0x7f8608007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.632 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.629+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f861004e4e0 con 0x7f861406b3d0 2023-09-30T01:58:14.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f861406b930 con 0x7f861406b3d0 2023-09-30T01:58:14.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f86067fc700 1 -- 172.21.15.151:0/3196834253 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8610015080 con 0x7f861406b3d0 2023-09-30T01:58:14.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:14.828 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 msgr2=0x7f85f8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 0x7f85f8062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8614144f10 tx=0x7f8608007eb0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 msgr2=0x7f86140ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f86140b1150 tx=0x7f8610017000 comp rx=0 tx=0).stop 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 shutdown_connections 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f85f8060550 0x7f85f8062a00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8614140c10 0x7f861413c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86140b0960 0x7f861413bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 --2- 172.21.15.151:0/3196834253 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f861406b3d0 0x7f86140ab030 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 >> 172.21.15.151:0/3196834253 conn(0x7f8614070510 msgr2=0x7f861413f950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:14.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 shutdown_connections 2023-09-30T01:58:14.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:14.825+0000 7f861c43a700 1 -- 172.21.15.151:0/3196834253 wait complete. 2023-09-30T01:58:17.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:17.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 Processor -- start 2023-09-30T01:58:17.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 -- start start 2023-09-30T01:58:17.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fad50067fe0 0x7fad5013b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad500ac070 con 0x7fad50137b50 2023-09-30T01:58:17.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5006e840 con 0x7fad5006e470 2023-09-30T01:58:17.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad500ad0e0 con 0x7fad50067fe0 2023-09-30T01:58:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4ed9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fad50137b50 0x7fad50137ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:46976/0 (socket says 172.21.15.151:46976) 2023-09-30T01:58:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad45d9b700 1 --2- 172.21.15.151:0/2908787322 >> v2:172.21.15.172:3300/0 conn(0x7fad50067fe0 0x7fad5013b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.953+0000 7fad4ed9d700 1 -- 172.21.15.151:0/2908787322 learned_addr learned my addr 172.21.15.151:0/2908787322 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4dd9b700 1 -- 172.21.15.151:0/2908787322 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682556243 0 0) 0x7fad500ac070 con 0x7fad50137b50 2023-09-30T01:58:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4dd9b700 1 -- 172.21.15.151:0/2908787322 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3c003570 con 0x7fad50137b50 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad45d9b700 1 -- 172.21.15.151:0/2908787322 >> v1:172.21.15.172:6789/0 conn(0x7fad5006e470 legacy=0x7fad5013b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad45d9b700 1 -- 172.21.15.151:0/2908787322 >> v1:172.21.15.151:6789/0 conn(0x7fad50137b50 legacy=0x7fad50137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad45d9b700 1 -- 172.21.15.151:0/2908787322 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5013bdb0 con 0x7fad50067fe0 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad45d9b700 1 --2- 172.21.15.151:0/2908787322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50067fe0 0x7fad5013b870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fad380098e0 tx=0x7fad380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd6bbde76388f7f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4dd9b700 1 -- 172.21.15.151:0/2908787322 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3800b640 con 0x7fad50067fe0 2023-09-30T01:58:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4dd9b700 1 -- 172.21.15.151:0/2908787322 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3800b7a0 con 0x7fad50067fe0 2023-09-30T01:58:17.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4dd9b700 1 -- 172.21.15.151:0/2908787322 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3800bb20 con 0x7fad50067fe0 2023-09-30T01:58:17.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 -- 172.21.15.151:0/2908787322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50067fe0 msgr2=0x7fad5013b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 --2- 172.21.15.151:0/2908787322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50067fe0 0x7fad5013b870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fad380098e0 tx=0x7fad380091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 -- 172.21.15.151:0/2908787322 shutdown_connections 2023-09-30T01:58:17.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 --2- 172.21.15.151:0/2908787322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50067fe0 0x7fad5013b870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 -- 172.21.15.151:0/2908787322 >> 172.21.15.151:0/2908787322 conn(0x7fad50070510 msgr2=0x7fad50070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:17.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 -- 172.21.15.151:0/2908787322 shutdown_connections 2023-09-30T01:58:17.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.957+0000 7fad4f59e700 1 -- 172.21.15.151:0/2908787322 wait complete. 2023-09-30T01:58:17.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 Processor -- start 2023-09-30T01:58:17.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- start start 2023-09-30T01:58:17.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50137b50 0x7fad5013cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5013fed0 con 0x7fad5006e470 2023-09-30T01:58:17.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5013d130 con 0x7fad50137b50 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad50140360 con 0x7fad50067fe0 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad45d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad45d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49296/0 (socket says 172.21.15.151:49296) 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:58680/0 (socket says 172.21.15.151:58680) 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad45d9b700 1 -- 172.21.15.151:0/649738435 learned_addr learned my addr 172.21.15.151:0/649738435 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:17.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4ed9d700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50137b50 0x7fad5013cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 -- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50137b50 msgr2=0x7fad5013cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50137b50 0x7fad5013cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 -- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 msgr2=0x7fad5013c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 -- 172.21.15.151:0/649738435 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38009420 con 0x7fad50067fe0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad45d9b700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4e59c700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c590 tx=0x7fad40007e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011590 con 0x7fad50067fe0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40011710 con 0x7fad50067fe0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad500a5760 con 0x7fad50067fe0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011ad0 con 0x7fad50067fe0 2023-09-30T01:58:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.961+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5013e480 con 0x7fad50067fe0 2023-09-30T01:58:17.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.965+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad500008d0 con 0x7fad50067fe0 2023-09-30T01:58:17.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.965+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fad4001c460 con 0x7fad50067fe0 2023-09-30T01:58:17.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.965+0000 7fad46ffd700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 0x7fad30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.965+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fad4007bd30 con 0x7fad50067fe0 2023-09-30T01:58:17.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.965+0000 7fad45d9b700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 0x7fad30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.969+0000 7fad45d9b700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 0x7fad30062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad5013eb50 tx=0x7fad38017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:17.973+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad4004c510 con 0x7fad50067fe0 2023-09-30T01:58:18.171 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad50138bb0 con 0x7fad50067fe0 2023-09-30T01:58:18.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad46ffd700 1 -- 172.21.15.151:0/649738435 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fad4004fb30 con 0x7fad50067fe0 2023-09-30T01:58:18.172 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:18.172 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:18.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 msgr2=0x7fad30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:18.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 0x7fad30062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad5013eb50 tx=0x7fad38017000 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 msgr2=0x7fad5013f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c590 tx=0x7fad40007e80 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 shutdown_connections 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fad30060550 0x7fad30062a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad50137b50 0x7fad5013cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad5006e470 0x7fad5013c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 --2- 172.21.15.151:0/649738435 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad50067fe0 0x7fad5013f2f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 >> 172.21.15.151:0/649738435 conn(0x7fad50070510 msgr2=0x7fad500708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 shutdown_connections 2023-09-30T01:58:18.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:18.169+0000 7fad4f59e700 1 -- 172.21.15.151:0/649738435 wait complete. 2023-09-30T01:58:21.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:21.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.293+0000 7f3365017700 1 Processor -- start 2023-09-30T01:58:21.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.293+0000 7f3365017700 1 -- start start 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f33600708e0 0x7f336013e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 0x7f3360137050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600a6d90 con 0x7f3360070510 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336013e660 con 0x7f33600708e0 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600a7070 con 0x7f33600ac830 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 0x7f3360137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335ed9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f3360070510 0x7f336013da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:49352/0 (socket says 172.21.15.151:49352) 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335ed9d700 1 -- 172.21.15.151:0/1455807503 learned_addr learned my addr 172.21.15.151:0/1455807503 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:21.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3357fff700 1 --2- 172.21.15.151:0/1455807503 >> v2:172.21.15.172:3300/0 conn(0x7f33600708e0 0x7f336013e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 -- 172.21.15.151:0/1455807503 >> v2:172.21.15.172:3300/0 conn(0x7f33600708e0 msgr2=0x7f336013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:21.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 --2- 172.21.15.151:0/1455807503 >> v2:172.21.15.172:3300/0 conn(0x7f33600708e0 0x7f336013e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 -- 172.21.15.151:0/1455807503 >> v1:172.21.15.151:6789/0 conn(0x7f3360070510 legacy=0x7f336013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 -- 172.21.15.151:0/1455807503 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336006ea10 con 0x7f33600ac830 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335f59e700 1 --2- 172.21.15.151:0/1455807503 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 0x7f3360137050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e60 tx=0x7f3358009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503a16d16a7925f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335e59c700 1 -- 172.21.15.151:0/1455807503 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335800e070 con 0x7f33600ac830 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335e59c700 1 -- 172.21.15.151:0/1455807503 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335800bc30 con 0x7f33600ac830 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f335e59c700 1 -- 172.21.15.151:0/1455807503 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358012430 con 0x7f33600ac830 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- 172.21.15.151:0/1455807503 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 msgr2=0x7f3360137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 --2- 172.21.15.151:0/1455807503 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 0x7f3360137050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e60 tx=0x7f3358009780 comp rx=0 tx=0).stop 2023-09-30T01:58:21.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- 172.21.15.151:0/1455807503 shutdown_connections 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 --2- 172.21.15.151:0/1455807503 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600ac830 0x7f3360137050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 --2- 172.21.15.151:0/1455807503 >> v2:172.21.15.172:3300/0 conn(0x7f33600708e0 0x7f336013e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- 172.21.15.151:0/1455807503 >> 172.21.15.151:0/1455807503 conn(0x7f3360067510 msgr2=0x7f3360067910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- 172.21.15.151:0/1455807503 shutdown_connections 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.297+0000 7f3365017700 1 -- 172.21.15.151:0/1455807503 wait complete. 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 Processor -- start 2023-09-30T01:58:21.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- start start 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600708e0 0x7f336014ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33600ac830 0x7f336014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600ae5d0 con 0x7f3360070510 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336006e7a0 con 0x7f33600ac830 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336014b8f0 con 0x7f33600708e0 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3357fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600708e0 0x7f336014ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54038/0 (socket says 172.21.15.151:54038) 2023-09-30T01:58:21.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 -- 172.21.15.151:0/1674580457 learned_addr learned my addr 172.21.15.151:0/1674580457 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:21.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335f59e700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33600ac830 0x7f336014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 -- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600708e0 msgr2=0x7f336014ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600708e0 0x7f336014ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 -- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33600ac830 msgr2=0x7f336014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33600ac830 0x7f336014b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 -- 172.21.15.151:0/1674580457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3358009a20 con 0x7f3360070510 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f335ed9d700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f335000c7c0 tx=0x7f3350007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350005b30 con 0x7f3360070510 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3350005c90 con 0x7f3360070510 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336013a220 con 0x7f3360070510 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350007320 con 0x7f3360070510 2023-09-30T01:58:21.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.301+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336013aef0 con 0x7f3360070510 2023-09-30T01:58:21.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.305+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336013c7f0 con 0x7f3360070510 2023-09-30T01:58:21.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.305+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3350007690 con 0x7f3360070510 2023-09-30T01:58:21.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.305+0000 7f3356ffd700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 0x7f3340062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:21.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.305+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f335007ba40 con 0x7f3360070510 2023-09-30T01:58:21.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.305+0000 7f3357fff700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 0x7f3340062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:21.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.309+0000 7f3357fff700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 0x7f3340062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f336013bdb0 tx=0x7f3348008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:21.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.313+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f336013c7f0 con 0x7f3360070510 2023-09-30T01:58:21.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f336013a3b0 con 0x7f3360070510 2023-09-30T01:58:21.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3356ffd700 1 -- 172.21.15.151:0/1674580457 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f336013a3b0 con 0x7f3360070510 2023-09-30T01:58:21.508 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:21.509 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 msgr2=0x7f3340062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 0x7f3340062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f336013bdb0 tx=0x7f3348008040 comp rx=0 tx=0).stop 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 msgr2=0x7f336006dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.505+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f335000c7c0 tx=0x7f3350007f30 comp rx=0 tx=0).stop 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 shutdown_connections 2023-09-30T01:58:21.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3340060550 0x7f3340062a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33600ac830 0x7f336014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f33600708e0 0x7f336014ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 --2- 172.21.15.151:0/1674580457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3360070510 0x7f336006dd70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 >> 172.21.15.151:0/1674580457 conn(0x7f3360067510 msgr2=0x7f336013cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 shutdown_connections 2023-09-30T01:58:21.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:21.509+0000 7f3365017700 1 -- 172.21.15.151:0/1674580457 wait complete. 2023-09-30T01:58:24.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:24.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.629+0000 7f38ae9a8700 1 Processor -- start 2023-09-30T01:58:24.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.629+0000 7f38ae9a8700 1 -- start start 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f38a80ae600 0x7f38a813b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38a80ac810 0x7f38a813b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80addd0 con 0x7f38a813e5b0 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80acc60 con 0x7f38a80ac810 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80ae9d0 con 0x7f38a80ae600 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38a80ac810 0x7f38a813b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38a80ac810 0x7f38a813b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54046/0 (socket says 172.21.15.151:54046) 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 -- 172.21.15.151:0/4117356772 learned_addr learned my addr 172.21.15.151:0/4117356772 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad9a6700 1 --2- 172.21.15.151:0/4117356772 >> v2:172.21.15.172:3300/0 conn(0x7f38a80ae600 0x7f38a813b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 -- 172.21.15.151:0/4117356772 >> v2:172.21.15.172:3300/0 conn(0x7f38a80ae600 msgr2=0x7f38a813b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 --2- 172.21.15.151:0/4117356772 >> v2:172.21.15.172:3300/0 conn(0x7f38a80ae600 0x7f38a813b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 -- 172.21.15.151:0/4117356772 >> v1:172.21.15.172:6789/0 conn(0x7f38a813e5b0 legacy=0x7f38a813ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 -- 172.21.15.151:0/4117356772 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80acf40 con 0x7f38a80ac810 2023-09-30T01:58:24.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ad1a5700 1 --2- 172.21.15.151:0/4117356772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a80ac810 0x7f38a813b840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38a80713c0 tx=0x7f38a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3630eb50b2cab1c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ac9a4700 1 -- 172.21.15.151:0/4117356772 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a400e070 con 0x7f38a80ac810 2023-09-30T01:58:24.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ac9a4700 1 -- 172.21.15.151:0/4117356772 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a400ba20 con 0x7f38a80ac810 2023-09-30T01:58:24.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ac9a4700 1 -- 172.21.15.151:0/4117356772 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a40092e0 con 0x7f38a80ac810 2023-09-30T01:58:24.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- 172.21.15.151:0/4117356772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a80ac810 msgr2=0x7f38a813b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/4117356772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a80ac810 0x7f38a813b840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38a80713c0 tx=0x7f38a4009840 comp rx=0 tx=0).stop 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- 172.21.15.151:0/4117356772 shutdown_connections 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/4117356772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a80ac810 0x7f38a813b840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/4117356772 >> v2:172.21.15.172:3300/0 conn(0x7f38a80ae600 0x7f38a813b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- 172.21.15.151:0/4117356772 >> 172.21.15.151:0/4117356772 conn(0x7f38a8070510 msgr2=0x7f38a8070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- 172.21.15.151:0/4117356772 shutdown_connections 2023-09-30T01:58:24.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.633+0000 7f38ae9a8700 1 -- 172.21.15.151:0/4117356772 wait complete. 2023-09-30T01:58:24.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 Processor -- start 2023-09-30T01:58:24.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- start start 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38a80ac810 0x7f38a814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38a80ae600 0x7f38a813a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a814e970 con 0x7f38a813e5b0 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a814ec20 con 0x7f38a80ac810 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80a6620 con 0x7f38a80ae600 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54058/0 (socket says 172.21.15.151:54058) 2023-09-30T01:58:24.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 -- 172.21.15.151:0/2645051527 learned_addr learned my addr 172.21.15.151:0/2645051527 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:24.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ad9a6700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38a80ac810 0x7f38a814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ad1a5700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38a80ae600 0x7f38a813a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 -- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38a80ae600 msgr2=0x7f38a813a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38a80ae600 0x7f38a813a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 -- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38a80ac810 msgr2=0x7f38a814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38a80ac810 0x7f38a814e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 -- 172.21.15.151:0/2645051527 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a4009ae0 con 0x7f38a813e5b0 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae1a7700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c4f0 tx=0x7f38a0007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a000ce60 con 0x7f38a813e5b0 2023-09-30T01:58:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0011590 con 0x7f38a813e5b0 2023-09-30T01:58:24.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a813a9d0 con 0x7f38a813e5b0 2023-09-30T01:58:24.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0011930 con 0x7f38a813e5b0 2023-09-30T01:58:24.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.637+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a8134d10 con 0x7f38a813e5b0 2023-09-30T01:58:24.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.641+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a80b17c0 con 0x7f38a813e5b0 2023-09-30T01:58:24.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.641+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f38a0011ca0 con 0x7f38a813e5b0 2023-09-30T01:58:24.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.641+0000 7f389dffb700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 0x7f388c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.641+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f38a007b8e0 con 0x7f38a813e5b0 2023-09-30T01:58:24.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.641+0000 7f38ad9a6700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 0x7f388c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.645+0000 7f38ad9a6700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 0x7f388c062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3898009db0 tx=0x7f38980091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.649+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38a80b17c0 con 0x7f38a813e5b0 2023-09-30T01:58:24.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.849+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38a80b17c0 con 0x7f38a813e5b0 2023-09-30T01:58:24.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.849+0000 7f389dffb700 1 -- 172.21.15.151:0/2645051527 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f38a80b17c0 con 0x7f38a813e5b0 2023-09-30T01:58:24.854 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:24.854 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:24.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 msgr2=0x7f388c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 0x7f388c062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3898009db0 tx=0x7f38980091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 msgr2=0x7f38a81347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c4f0 tx=0x7f38a0007f50 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 shutdown_connections 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f388c060550 0x7f388c062a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38a813e5b0 0x7f38a81347d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38a80ae600 0x7f38a813a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 --2- 172.21.15.151:0/2645051527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38a80ac810 0x7f38a814e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 >> 172.21.15.151:0/2645051527 conn(0x7f38a8070510 msgr2=0x7f38a80b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 shutdown_connections 2023-09-30T01:58:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:24.853+0000 7f38ae9a8700 1 -- 172.21.15.151:0/2645051527 wait complete. 2023-09-30T01:58:27.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:27.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 Processor -- start 2023-09-30T01:58:27.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 -- start start 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbf10070740 0x7fbf100a4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100a7030 con 0x7fbf1013c450 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100a6b80 con 0x7fbf100ac480 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.973+0000 7fbf16501700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100a5090 con 0x7fbf10070740 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0ffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbf10070740 0x7fbf100a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf14a9e700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbf1013c450 0x7fbf1013c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:49364/0 (socket says 172.21.15.151:49364) 2023-09-30T01:58:27.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf14a9e700 1 -- 172.21.15.151:0/1933522455 learned_addr learned my addr 172.21.15.151:0/1933522455 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:27.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0ffff700 1 -- 172.21.15.151:0/1933522455 >> v1:172.21.15.172:6789/0 conn(0x7fbf100ac480 legacy=0x7fbf100a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0ffff700 1 -- 172.21.15.151:0/1933522455 >> v1:172.21.15.151:6789/0 conn(0x7fbf1013c450 legacy=0x7fbf1013c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0effd700 1 -- 172.21.15.151:0/1933522455 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966754475 0 0) 0x7fbf100a7030 con 0x7fbf1013c450 2023-09-30T01:58:27.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0ffff700 1 -- 172.21.15.151:0/1933522455 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100a6d50 con 0x7fbf10070740 2023-09-30T01:58:27.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0ffff700 1 --2- 172.21.15.151:0/1933522455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100a4440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098e0 tx=0x7fbef80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a4bf2ad174b5ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0effd700 1 -- 172.21.15.151:0/1933522455 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8009e80 con 0x7fbf10070740 2023-09-30T01:58:27.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0effd700 1 -- 172.21.15.151:0/1933522455 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef800b860 con 0x7fbf10070740 2023-09-30T01:58:27.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf0effd700 1 -- 172.21.15.151:0/1933522455 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef800bbe0 con 0x7fbf10070740 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 -- 172.21.15.151:0/1933522455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 msgr2=0x7fbf100a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 --2- 172.21.15.151:0/1933522455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100a4440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098e0 tx=0x7fbef80091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 -- 172.21.15.151:0/1933522455 shutdown_connections 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 --2- 172.21.15.151:0/1933522455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100a4440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 -- 172.21.15.151:0/1933522455 >> 172.21.15.151:0/1933522455 conn(0x7fbf10067710 msgr2=0x7fbf10067b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 -- 172.21.15.151:0/1933522455 shutdown_connections 2023-09-30T01:58:27.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.977+0000 7fbf16501700 1 -- 172.21.15.151:0/1933522455 wait complete. 2023-09-30T01:58:27.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 Processor -- start 2023-09-30T01:58:27.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- start start 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf1013c450 0x7fbf100a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100a8750 con 0x7fbf10070740 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100a8430 con 0x7fbf1013c450 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100a9f90 con 0x7fbf100ac480 2023-09-30T01:58:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54068/0 (socket says 172.21.15.151:54068) 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49484/0 (socket says 172.21.15.151:49484) 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0ffff700 1 -- 172.21.15.151:0/579709112 learned_addr learned my addr 172.21.15.151:0/579709112 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf14a9e700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf1013c450 0x7fbf100a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 -- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf1013c450 msgr2=0x7fbf100a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf1013c450 0x7fbf100a8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 -- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 msgr2=0x7fbf100aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 -- 172.21.15.151:0/579709112 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009420 con 0x7fbf100ac480 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0ffff700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf0f7fe700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000c910 tx=0x7fbf000016f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00012630 con 0x7fbf100ac480 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf000127b0 con 0x7fbf100ac480 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf100aa270 con 0x7fbf100ac480 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00012b70 con 0x7fbf100ac480 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.981+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1006ce30 con 0x7fbf100ac480 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100a88c0 con 0x7fbf100ac480 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fbf00003030 con 0x7fbf100ac480 2023-09-30T01:58:27.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf07fff700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 0x7fbef0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fbf0007c490 con 0x7fbf100ac480 2023-09-30T01:58:27.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf0ffff700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 0x7fbef0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.985+0000 7fbf0ffff700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 0x7fbef0062a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbef80091e0 tx=0x7fbef8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:27.993+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf0004cc70 con 0x7fbf100ac480 2023-09-30T01:58:28.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.189+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf100670a0 con 0x7fbf100ac480 2023-09-30T01:58:28.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.189+0000 7fbf07fff700 1 -- 172.21.15.151:0/579709112 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbf00050290 con 0x7fbf100ac480 2023-09-30T01:58:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:28.195 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 msgr2=0x7fbef0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 0x7fbef0062a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbef80091e0 tx=0x7fbef8017000 comp rx=0 tx=0).stop 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 msgr2=0x7fbf100a7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000c910 tx=0x7fbf000016f0 comp rx=0 tx=0).stop 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 shutdown_connections 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbef0060550 0x7fbef0062a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf1013c450 0x7fbf100a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:28.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf100ac480 0x7fbf100a7ef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:28.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 --2- 172.21.15.151:0/579709112 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf10070740 0x7fbf100aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:28.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 >> 172.21.15.151:0/579709112 conn(0x7fbf10067710 msgr2=0x7fbf100ae550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:28.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 shutdown_connections 2023-09-30T01:58:28.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:28.193+0000 7fbf16501700 1 -- 172.21.15.151:0/579709112 wait complete. 2023-09-30T01:58:31.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 Processor -- start 2023-09-30T01:58:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 -- start start 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f614c139fb0 0x7f614c13a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c070b10 con 0x7f614c070740 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614c0ac280 con 0x7f614c139fb0 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0ac560 con 0x7f614c06f140 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38262/0 (socket says 172.21.15.151:38262) 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 -- 172.21.15.151:0/174025204 learned_addr learned my addr 172.21.15.151:0/174025204 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6150fb3700 1 --2- 172.21.15.151:0/174025204 >> v2:172.21.15.151:3300/0 conn(0x7f614c139fb0 0x7f614c13a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 -- 172.21.15.151:0/174025204 >> v2:172.21.15.151:3300/0 conn(0x7f614c139fb0 msgr2=0x7f614c13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 --2- 172.21.15.151:0/174025204 >> v2:172.21.15.151:3300/0 conn(0x7f614c139fb0 0x7f614c13a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 -- 172.21.15.151:0/174025204 >> v1:172.21.15.172:6789/0 conn(0x7f614c070740 legacy=0x7f614c0aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 -- 172.21.15.151:0/174025204 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c06ec00 con 0x7f614c06f140 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f6143fff700 1 --2- 172.21.15.151:0/174025204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9b8fab6b800cdc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f614b7fe700 1 -- 172.21.15.151:0/174025204 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00b820 con 0x7f614c06f140 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f614b7fe700 1 -- 172.21.15.151:0/174025204 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c00b980 con 0x7f614c06f140 2023-09-30T01:58:31.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f614b7fe700 1 -- 172.21.15.151:0/174025204 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00bd40 con 0x7f614c06f140 2023-09-30T01:58:31.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 -- 172.21.15.151:0/174025204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 msgr2=0x7f614c13d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.317+0000 7f61517b4700 1 --2- 172.21.15.151:0/174025204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f613c0099b0 tx=0x7f613c009250 comp rx=0 tx=0).stop 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 -- 172.21.15.151:0/174025204 shutdown_connections 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 --2- 172.21.15.151:0/174025204 >> v2:172.21.15.151:3300/0 conn(0x7f614c139fb0 0x7f614c13a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 --2- 172.21.15.151:0/174025204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c13d2d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 -- 172.21.15.151:0/174025204 >> 172.21.15.151:0/174025204 conn(0x7f614c067510 msgr2=0x7f614c067910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 -- 172.21.15.151:0/174025204 shutdown_connections 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 -- 172.21.15.151:0/174025204 wait complete. 2023-09-30T01:58:31.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 Processor -- start 2023-09-30T01:58:31.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.321+0000 7f61517b4700 1 -- start start 2023-09-30T01:58:31.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c06db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f614c070740 0x7f614c13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c140e00 con 0x7f614c139fb0 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1187d0 con 0x7f614c070740 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c140ab0 con 0x7f614c06f140 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:43600/0 (socket says 172.21.15.151:43600) 2023-09-30T01:58:31.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f614bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 -- 172.21.15.151:0/3827724598 learned_addr learned my addr 172.21.15.151:0/3827724598 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6143fff700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f614c070740 0x7f614c13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 -- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 msgr2=0x7f614c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 -- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f614c070740 msgr2=0x7f614c13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f614c070740 0x7f614c13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 -- 172.21.15.151:0/3827724598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0094f0 con 0x7f614c139fb0 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6150fb3700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6144009fa0 tx=0x7f6144009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400ebc0 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614400ed20 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c118ab0 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144014670 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c1404c0 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.325+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c140fb0 con 0x7f614c139fb0 2023-09-30T01:58:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.329+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6144014a20 con 0x7f614c139fb0 2023-09-30T01:58:31.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.329+0000 7f6148ff9700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 0x7f612c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.329+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f614407cc20 con 0x7f614c139fb0 2023-09-30T01:58:31.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.329+0000 7f614bfff700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 0x7f612c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.329+0000 7f614bfff700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 0x7f612c062a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6134003a30 tx=0x7f6134007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.337+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f614c140fb0 con 0x7f614c139fb0 2023-09-30T01:58:31.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.537+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f614c140fb0 con 0x7f614c139fb0 2023-09-30T01:58:31.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.537+0000 7f6148ff9700 1 -- 172.21.15.151:0/3827724598 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f614c140fb0 con 0x7f614c139fb0 2023-09-30T01:58:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:31.540 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 msgr2=0x7f612c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 0x7f612c062a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6134003a30 tx=0x7f6134007ea0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 msgr2=0x7f614c150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6144009fa0 tx=0x7f6144009900 comp rx=0 tx=0).stop 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 shutdown_connections 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f612c060580 0x7f612c062a30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f614c139fb0 0x7f614c150430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f614c070740 0x7f614c13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 --2- 172.21.15.151:0/3827724598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f614c06f140 0x7f614c06db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 >> 172.21.15.151:0/3827724598 conn(0x7f614c067510 msgr2=0x7f614c0ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 shutdown_connections 2023-09-30T01:58:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:31.541+0000 7f61517b4700 1 -- 172.21.15.151:0/3827724598 wait complete. 2023-09-30T01:58:34.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:34.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 Processor -- start 2023-09-30T01:58:34.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 -- start start 2023-09-30T01:58:34.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d806d8b0 con 0x7f16d806b320 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d806db70 con 0x7f16d806af50 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16dcb15700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8139e10 con 0x7f16d81399b0 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38278/0 (socket says 172.21.15.151:38278) 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16cdd9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f16d806b320 0x7f16d806e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41764/0 (socket says 172.21.15.151:41764) 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 -- 172.21.15.151:0/2334264620 learned_addr learned my addr 172.21.15.151:0/2334264620 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 -- 172.21.15.151:0/2334264620 >> v1:172.21.15.172:6789/0 conn(0x7f16d806af50 legacy=0x7f16d806be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 -- 172.21.15.151:0/2334264620 >> v1:172.21.15.151:6789/0 conn(0x7f16d806b320 legacy=0x7f16d806e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d5d9b700 1 -- 172.21.15.151:0/2334264620 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125233679 0 0) 0x7f16d806d8b0 con 0x7f16d806b320 2023-09-30T01:58:34.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.669+0000 7f16d6d9d700 1 -- 172.21.15.151:0/2334264620 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d806ced0 con 0x7f16d81399b0 2023-09-30T01:58:34.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d6d9d700 1 --2- 172.21.15.151:0/2334264620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009f50 tx=0x7f16d0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d861bc6d0fafacf3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d5d9b700 1 -- 172.21.15.151:0/2334264620 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d000e070 con 0x7f16d81399b0 2023-09-30T01:58:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d5d9b700 1 -- 172.21.15.151:0/2334264620 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d000bab0 con 0x7f16d81399b0 2023-09-30T01:58:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d5d9b700 1 -- 172.21.15.151:0/2334264620 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d00092e0 con 0x7f16d81399b0 2023-09-30T01:58:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- 172.21.15.151:0/2334264620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 msgr2=0x7f16d80a9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 --2- 172.21.15.151:0/2334264620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009f50 tx=0x7f16d0009840 comp rx=0 tx=0).stop 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- 172.21.15.151:0/2334264620 shutdown_connections 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 --2- 172.21.15.151:0/2334264620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d81399b0 0x7f16d80a9490 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- 172.21.15.151:0/2334264620 >> 172.21.15.151:0/2334264620 conn(0x7f16d8070fe0 msgr2=0x7f16d80af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- 172.21.15.151:0/2334264620 shutdown_connections 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- 172.21.15.151:0/2334264620 wait complete. 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 Processor -- start 2023-09-30T01:58:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- start start 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16d806af50 0x7f16d8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 0x7f16d8118ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8118a50 con 0x7f16d806af50 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d811a950 con 0x7f16d81399b0 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16dcb15700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81513d0 con 0x7f16d806b320 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16d806af50 0x7f16d8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38286/0 (socket says 172.21.15.151:38286) 2023-09-30T01:58:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 -- 172.21.15.151:0/3757956525 learned_addr learned my addr 172.21.15.151:0/3757956525 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d6d9d700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 0x7f16d8118ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 -- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 msgr2=0x7f16d8118ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 0x7f16d8118ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 -- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16d806af50 msgr2=0x7f16d8150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16d806af50 0x7f16d8150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 -- 172.21.15.151:0/3757956525 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0009ae0 con 0x7f16d806b320 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16d6d9d700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 0x7f16d8118ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.673+0000 7f16cdd9b700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c840 tx=0x7f16c0007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0005b40 con 0x7f16d806b320 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c0005ca0 con 0x7f16d806b320 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d8151590 con 0x7f16d806b320 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0007320 con 0x7f16d806b320 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d813c420 con 0x7f16d806b320 2023-09-30T01:58:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d8118c00 con 0x7f16d806b320 2023-09-30T01:58:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f16c001b3b0 con 0x7f16d806b320 2023-09-30T01:58:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 0x7f16b8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f16c007bb10 con 0x7f16d806b320 2023-09-30T01:58:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.677+0000 7f16d659c700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 0x7f16b8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.681+0000 7f16d659c700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 0x7f16b8062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16c80039e0 tx=0x7f16c8007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.685+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f16d8118c00 con 0x7f16d806b320 2023-09-30T01:58:34.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16d8118c00 con 0x7f16d806b320 2023-09-30T01:58:34.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16ceffd700 1 -- 172.21.15.151:0/3757956525 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f16c0017100 con 0x7f16d806b320 2023-09-30T01:58:34.884 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:34.884 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 msgr2=0x7f16b8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 0x7f16b8062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16c80039e0 tx=0x7f16c8007ec0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 msgr2=0x7f16d81182f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c840 tx=0x7f16c0007d90 comp rx=0 tx=0).stop 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 shutdown_connections 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f16b8060550 0x7f16b8062a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16d81399b0 0x7f16d8118ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16d806b320 0x7f16d81182f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 --2- 172.21.15.151:0/3757956525 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16d806af50 0x7f16d8150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 >> 172.21.15.151:0/3757956525 conn(0x7f16d8070fe0 msgr2=0x7f16d806b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 shutdown_connections 2023-09-30T01:58:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:34.881+0000 7f16dcb15700 1 -- 172.21.15.151:0/3757956525 wait complete. 2023-09-30T01:58:37.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:38.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 Processor -- start 2023-09-30T01:58:38.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- start start 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9df4070740 0x7f9df406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df406c350 con 0x7f9df4142080 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df4070b10 con 0x7f9df4070740 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df406d0a0 con 0x7f9df4141730 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38298/0 (socket says 172.21.15.151:38298) 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 -- 172.21.15.151:0/856502551 learned_addr learned my addr 172.21.15.151:0/856502551 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9b22700 1 --2- 172.21.15.151:0/856502551 >> v2:172.21.15.172:3300/0 conn(0x7f9df4070740 0x7f9df406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 -- 172.21.15.151:0/856502551 >> v2:172.21.15.172:3300/0 conn(0x7f9df4070740 msgr2=0x7f9df406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 --2- 172.21.15.151:0/856502551 >> v2:172.21.15.172:3300/0 conn(0x7f9df4070740 0x7f9df406b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 -- 172.21.15.151:0/856502551 >> v1:172.21.15.172:6789/0 conn(0x7f9df4142080 legacy=0x7f9df4144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 -- 172.21.15.151:0/856502551 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df406d380 con 0x7f9df4141730 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df9321700 1 --2- 172.21.15.151:0/856502551 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9de80099b0 tx=0x7f9de8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2207bf59cd63ff6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df8b20700 1 -- 172.21.15.151:0/856502551 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800b860 con 0x7f9df4141730 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df8b20700 1 -- 172.21.15.151:0/856502551 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800b9c0 con 0x7f9df4141730 2023-09-30T01:58:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9df8b20700 1 -- 172.21.15.151:0/856502551 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800bd60 con 0x7f9df4141730 2023-09-30T01:58:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- 172.21.15.151:0/856502551 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 msgr2=0x7f9df4141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 --2- 172.21.15.151:0/856502551 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9de80099b0 tx=0x7f9de8009250 comp rx=0 tx=0).stop 2023-09-30T01:58:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- 172.21.15.151:0/856502551 shutdown_connections 2023-09-30T01:58:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 --2- 172.21.15.151:0/856502551 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4141730 0x7f9df4141b40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 --2- 172.21.15.151:0/856502551 >> v2:172.21.15.172:3300/0 conn(0x7f9df4070740 0x7f9df406b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- 172.21.15.151:0/856502551 >> 172.21.15.151:0/856502551 conn(0x7f9df40b03f0 msgr2=0x7f9df40b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- 172.21.15.151:0/856502551 shutdown_connections 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.009+0000 7f9dfab24700 1 -- 172.21.15.151:0/856502551 wait complete. 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 Processor -- start 2023-09-30T01:58:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 -- start start 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 0x7f9df40a80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 0x7f9df40a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40a7c20 con 0x7f9df4141730 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40a9af0 con 0x7f9df4142080 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfab24700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40a9d60 con 0x7f9df4070740 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38300/0 (socket says 172.21.15.151:38300) 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 -- 172.21.15.151:0/1498008011 learned_addr learned my addr 172.21.15.151:0/1498008011 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9321700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 0x7f9df40a80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 -- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 msgr2=0x7f9df40a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfa323700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 0x7f9df40a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 0x7f9df40a85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 -- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 msgr2=0x7f9df40a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 0x7f9df40a80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 -- 172.21.15.151:0/1498008011 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80094f0 con 0x7f9df4070740 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9321700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 0x7f9df40a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9dfa323700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 0x7f9df40a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9df9b22700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d890 tx=0x7f9df0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.013+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0009610 con 0x7f9df4070740 2023-09-30T01:58:38.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0009770 con 0x7f9df4070740 2023-09-30T01:58:38.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df40a8b30 con 0x7f9df4070740 2023-09-30T01:58:38.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0009b30 con 0x7f9df4070740 2023-09-30T01:58:38.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df4151b90 con 0x7f9df4070740 2023-09-30T01:58:38.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df40008d0 con 0x7f9df4070740 2023-09-30T01:58:38.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9df001e460 con 0x7f9df4070740 2023-09-30T01:58:38.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.017+0000 7f9de5ffb700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 0x7f9dd8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:38.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.021+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9df007ccf0 con 0x7f9df4070740 2023-09-30T01:58:38.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.021+0000 7f9df9321700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 0x7f9dd8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:38.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.021+0000 7f9df9321700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 0x7f9dd8062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de80099b0 tx=0x7f9de8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:38.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.025+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9df004d430 con 0x7f9df4070740 2023-09-30T01:58:38.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.217+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9df406be30 con 0x7f9df4070740 2023-09-30T01:58:38.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.217+0000 7f9de5ffb700 1 -- 172.21.15.151:0/1498008011 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9df0050a50 con 0x7f9df4070740 2023-09-30T01:58:38.222 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:38.222 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 msgr2=0x7f9dd8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 0x7f9dd8062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de80099b0 tx=0x7f9de8017000 comp rx=0 tx=0).stop 2023-09-30T01:58:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 msgr2=0x7f9df4150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d890 tx=0x7f9df0008040 comp rx=0 tx=0).stop 2023-09-30T01:58:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 shutdown_connections 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9dd8060550 0x7f9dd8062a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9df4142080 0x7f9df40a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9df4141730 0x7f9df40a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 --2- 172.21.15.151:0/1498008011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9df4070740 0x7f9df4150ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 >> 172.21.15.151:0/1498008011 conn(0x7f9df40b03f0 msgr2=0x7f9df40ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 shutdown_connections 2023-09-30T01:58:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:38.221+0000 7f9dfab24700 1 -- 172.21.15.151:0/1498008011 wait complete. 2023-09-30T01:58:41.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:41.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 Processor -- start 2023-09-30T01:58:41.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- start start 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff0c406b180 0x7ff0c40ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c406c8e0 con 0x7ff0c406b550 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40abe70 con 0x7ff0c406b180 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40a9b00 con 0x7ff0c40ab460 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c8fc3700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff0c406b180 0x7ff0c40ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33376/0 (socket says 172.21.15.151:33376) 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 -- 172.21.15.151:0/3981602278 learned_addr learned my addr 172.21.15.151:0/3981602278 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 -- 172.21.15.151:0/3981602278 >> v2:172.21.15.151:3300/0 conn(0x7ff0c406b180 msgr2=0x7ff0c40ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 --2- 172.21.15.151:0/3981602278 >> v2:172.21.15.151:3300/0 conn(0x7ff0c406b180 0x7ff0c40ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 -- 172.21.15.151:0/3981602278 >> v1:172.21.15.172:6789/0 conn(0x7ff0c406b550 legacy=0x7ff0c4144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 -- 172.21.15.151:0/3981602278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4141b10 con 0x7ff0c40ab460 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c3fff700 1 --2- 172.21.15.151:0/3981602278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009f20 tx=0x7ff0bc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa62b84f64ddc873 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c37fe700 1 -- 172.21.15.151:0/3981602278 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc00e070 con 0x7ff0c40ab460 2023-09-30T01:58:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c37fe700 1 -- 172.21.15.151:0/3981602278 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc00b9d0 con 0x7ff0c40ab460 2023-09-30T01:58:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c37fe700 1 -- 172.21.15.151:0/3981602278 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0092e0 con 0x7ff0c40ab460 2023-09-30T01:58:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3981602278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 msgr2=0x7ff0c41415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3981602278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009f20 tx=0x7ff0bc009840 comp rx=0 tx=0).stop 2023-09-30T01:58:41.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3981602278 shutdown_connections 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3981602278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c40ab460 0x7ff0c41415d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3981602278 >> v2:172.21.15.151:3300/0 conn(0x7ff0c406b180 0x7ff0c40ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3981602278 >> 172.21.15.151:0/3981602278 conn(0x7ff0c4070510 msgr2=0x7ff0c4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3981602278 shutdown_connections 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.353+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3981602278 wait complete. 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 Processor -- start 2023-09-30T01:58:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- start start 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0c406b550 0x7ff0c40b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 0x7ff0c41547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40a92a0 con 0x7ff0c406b550 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40afcc0 con 0x7ff0c40ab460 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4154cf0 con 0x7ff0c406b180 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33392/0 (socket says 172.21.15.151:33392) 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 -- 172.21.15.151:0/3836669441 learned_addr learned my addr 172.21.15.151:0/3836669441 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:41.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c3fff700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0c406b550 0x7ff0c40b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c97c4700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 0x7ff0c41547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 -- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 msgr2=0x7ff0c41547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 0x7ff0c41547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 -- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0c406b550 msgr2=0x7ff0c40b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0c406b550 0x7ff0c40b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 -- 172.21.15.151:0/3836669441 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc009ae0 con 0x7ff0c406b180 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c97c4700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 0x7ff0c41547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c8fc3700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ecb0 tx=0x7ff0ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00c920 con 0x7ff0c406b180 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00ca80 con 0x7ff0c406b180 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40af1d0 con 0x7ff0c406b180 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00ce40 con 0x7ff0c406b180 2023-09-30T01:58:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.357+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c40af740 con 0x7ff0c406b180 2023-09-30T01:58:41.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.361+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c4154ec0 con 0x7ff0c406b180 2023-09-30T01:58:41.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.361+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff0ac013680 con 0x7ff0c406b180 2023-09-30T01:58:41.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.361+0000 7ff0c0ff9700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 0x7ff0a4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.361+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff0ac07c7f0 con 0x7ff0c406b180 2023-09-30T01:58:41.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.361+0000 7ff0c3fff700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 0x7ff0a4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.365+0000 7ff0c3fff700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 0x7ff0a4062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0c41415a0 tx=0x7ff0bc018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.369+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff0ac04cfd0 con 0x7ff0c406b180 2023-09-30T01:58:41.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.561+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff0c40afe30 con 0x7ff0c406b180 2023-09-30T01:58:41.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.561+0000 7ff0c0ff9700 1 -- 172.21.15.151:0/3836669441 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff0ac0505f0 con 0x7ff0c406b180 2023-09-30T01:58:41.566 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:41.566 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:41.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 msgr2=0x7ff0a4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 0x7ff0a4062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0c41415a0 tx=0x7ff0bc018000 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 msgr2=0x7ff0c4140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ecb0 tx=0x7ff0ac0091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 shutdown_connections 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7ff0a4060550 0x7ff0a4062a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0c40ab460 0x7ff0c41547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff0c406b550 0x7ff0c40b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 --2- 172.21.15.151:0/3836669441 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff0c406b180 0x7ff0c4140e70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 >> 172.21.15.151:0/3836669441 conn(0x7ff0c4070510 msgr2=0x7ff0c4145ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.565+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 shutdown_connections 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:41.569+0000 7ff0c9fc5700 1 -- 172.21.15.151:0/3836669441 wait complete. 2023-09-30T01:58:44.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:44.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 Processor -- start 2023-09-30T01:58:44.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 -- start start 2023-09-30T01:58:44.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f39580b0440 0x7f395806c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39580abe30 con 0x7f39580b0070 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395806d690 con 0x7f395806e070 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395d231700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395806d970 con 0x7f39580b0440 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f39580b0440 0x7f395806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f39580b0440 0x7f395806c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42490/0 (socket says 172.21.15.151:42490) 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 -- 172.21.15.151:0/4154951619 learned_addr learned my addr 172.21.15.151:0/4154951619 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f3956d9d700 1 --1- 172.21.15.151:0/4154951619 >> v1:172.21.15.151:6789/0 conn(0x7f39580b0070 0x7f395813a0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38790/0 (socket says 172.21.15.151:38790) 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 -- 172.21.15.151:0/4154951619 >> v1:172.21.15.172:6789/0 conn(0x7f395806e070 legacy=0x7f395806bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 -- 172.21.15.151:0/4154951619 >> v1:172.21.15.151:6789/0 conn(0x7f39580b0070 legacy=0x7f395813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.693+0000 7f395759e700 1 -- 172.21.15.151:0/4154951619 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395806dc50 con 0x7f39580b0440 2023-09-30T01:58:44.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395759e700 1 --2- 172.21.15.151:0/4154951619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f395806c670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f395000bc50 tx=0x7f395000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405f2a56ce2dc921 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f3955d9b700 1 -- 172.21.15.151:0/4154951619 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395000be60 con 0x7f39580b0440 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f3955d9b700 1 -- 172.21.15.151:0/4154951619 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395000de00 con 0x7f39580b0440 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f3955d9b700 1 -- 172.21.15.151:0/4154951619 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950019690 con 0x7f39580b0440 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- 172.21.15.151:0/4154951619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 msgr2=0x7f395806c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 --2- 172.21.15.151:0/4154951619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f395806c670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f395000bc50 tx=0x7f395000b4f0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- 172.21.15.151:0/4154951619 shutdown_connections 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 --2- 172.21.15.151:0/4154951619 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f395806c670 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- 172.21.15.151:0/4154951619 >> 172.21.15.151:0/4154951619 conn(0x7f3958070fe0 msgr2=0x7f39580af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- 172.21.15.151:0/4154951619 shutdown_connections 2023-09-30T01:58:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- 172.21.15.151:0/4154951619 wait complete. 2023-09-30T01:58:44.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 Processor -- start 2023-09-30T01:58:44.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.697+0000 7f395d231700 1 -- start start 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395806e070 0x7f39580a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 0x7f395813d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f39580a8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39580a81f0 con 0x7f39580b0440 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39580a6af0 con 0x7f395806e070 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39580a83f0 con 0x7f39580b0070 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395659c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 0x7f395813d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395659c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 0x7f395813d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33406/0 (socket says 172.21.15.151:33406) 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395659c700 1 -- 172.21.15.151:0/3199820836 learned_addr learned my addr 172.21.15.151:0/3199820836 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f39580a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f3956d9d700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395806e070 0x7f39580a6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 -- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 msgr2=0x7f395813d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 0x7f395813d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 -- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395806e070 msgr2=0x7f39580a6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395806e070 0x7f39580a6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 -- 172.21.15.151:0/3199820836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395000b790 con 0x7f39580b0440 2023-09-30T01:58:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395759e700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f39580a8810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f395806ac40 tx=0x7f395001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950006bc0 con 0x7f39580b0440 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3950010bb0 con 0x7f39580b0440 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950022420 con 0x7f39580b0440 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39580a62a0 con 0x7f39580b0440 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.701+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395813e5d0 con 0x7f39580b0440 2023-09-30T01:58:44.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39580670a0 con 0x7f39580b0440 2023-09-30T01:58:44.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f395002b430 con 0x7f39580b0440 2023-09-30T01:58:44.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f393effd700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 0x7f3944062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3950014070 con 0x7f39580b0440 2023-09-30T01:58:44.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f3956d9d700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 0x7f3944062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.705+0000 7f3956d9d700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 0x7f3944062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f394c004fd0 tx=0x7f394c007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.713+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f395004e750 con 0x7f39580b0440 2023-09-30T01:58:44.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.909+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f39580a6c30 con 0x7f39580b0440 2023-09-30T01:58:44.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.909+0000 7f393effd700 1 -- 172.21.15.151:0/3199820836 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3950027080 con 0x7f39580b0440 2023-09-30T01:58:44.913 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:44.913 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 msgr2=0x7f3944062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 0x7f3944062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f394c004fd0 tx=0x7f394c007eb0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 msgr2=0x7f39580a8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f39580a8810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f395806ac40 tx=0x7f395001b000 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 shutdown_connections 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3944060550 0x7f3944062a00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f39580b0440 0x7f39580a8810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f39580b0070 0x7f395813d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 --2- 172.21.15.151:0/3199820836 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f395806e070 0x7f39580a6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 >> 172.21.15.151:0/3199820836 conn(0x7f3958070fe0 msgr2=0x7f39580acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:44.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 shutdown_connections 2023-09-30T01:58:44.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:44.913+0000 7f395d231700 1 -- 172.21.15.151:0/3199820836 wait complete. 2023-09-30T01:58:47.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:48.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 Processor -- start 2023-09-30T01:58:48.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 -- start start 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5df00aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5df006e870 0x7f5df00a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df006ecd0 con 0x7f5df00ac510 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df00abbd0 con 0x7f5df006e870 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df5e8f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df013b6e0 con 0x7f5df00ac140 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5deffff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f5df00ac510 0x7f5df013afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38798/0 (socket says 172.21.15.151:38798) 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5deffff700 1 -- 172.21.15.151:0/3159130645 learned_addr learned my addr 172.21.15.151:0/3159130645 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df4e8d700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5df00aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 --2- 172.21.15.151:0/3159130645 >> v2:172.21.15.172:3300/0 conn(0x7f5df006e870 0x7f5df00a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688899702 0 0) 0x7f5df006ecd0 con 0x7f5df00ac510 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de0003570 con 0x7f5df00ac510 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738251644 0 0) 0x7f5de0003570 con 0x7f5df00ac510 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 msgr2=0x7f5df00aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5df00aa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 >> v2:172.21.15.172:3300/0 conn(0x7f5df006e870 msgr2=0x7f5df00a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 --2- 172.21.15.151:0/3159130645 >> v2:172.21.15.172:3300/0 conn(0x7f5df006e870 0x7f5df00a8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df0138c00 con 0x7f5df00ac510 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f5de4001490 con 0x7f5df00ac510 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 >> v1:172.21.15.151:6789/0 conn(0x7f5df00ac510 legacy=0x7f5df013afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5de0003a90 0x7f5de0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5de0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de0004d50 0x7f5de00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df00abbd0 con 0x7f5de0004d50 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df013b6e0 con 0x7f5de0003a90 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de000fe70 con 0x7f5df00ac140 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5deffff700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5de0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de0004d50 0x7f5de00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df4e8d700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5de0003a90 0x7f5de0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 -- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 msgr2=0x7f5de0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00ac140 0x7f5de0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 -- 172.21.15.151:0/3159130645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5de0003a90 msgr2=0x7f5de0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5de0003a90 0x7f5de0003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 -- 172.21.15.151:0/3159130645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0003570 con 0x7f5de0004d50 2023-09-30T01:58:48.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.037+0000 7f5df568e700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de0004d50 0x7f5de00051c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e210 tx=0x7f5de800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7495a3eb4624a468 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de800ea60 con 0x7f5de0004d50 2023-09-30T01:58:48.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de800f040 con 0x7f5de0004d50 2023-09-30T01:58:48.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5def7fe700 1 -- 172.21.15.151:0/3159130645 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de800ee10 con 0x7f5de0004d50 2023-09-30T01:58:48.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de0004d50 msgr2=0x7f5de00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/3159130645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de0004d50 0x7f5de00051c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e210 tx=0x7f5de800b040 comp rx=0 tx=0).stop 2023-09-30T01:58:48.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df568e700 1 -- 172.21.15.151:0/3159130645 reap_dead start 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- 172.21.15.151:0/3159130645 shutdown_connections 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- 172.21.15.151:0/3159130645 >> 172.21.15.151:0/3159130645 conn(0x7f5df0070510 msgr2=0x7f5df0070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- 172.21.15.151:0/3159130645 shutdown_connections 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- 172.21.15.151:0/3159130645 wait complete. 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 Processor -- start 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 -- start start 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.041+0000 7f5df5e8f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5df00a4580 0x7f5df00a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00a70c0 0x7f5df014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df00a7540 con 0x7f5df006d200 2023-09-30T01:58:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df00a8590 con 0x7f5df00a4580 2023-09-30T01:58:48.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df006d5d0 con 0x7f5df00a70c0 2023-09-30T01:58:48.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42518/0 (socket says 172.21.15.151:42518) 2023-09-30T01:58:48.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 -- 172.21.15.151:0/1250245398 learned_addr learned my addr 172.21.15.151:0/1250245398 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df568e700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00a70c0 0x7f5df014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5deffff700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5df00a4580 0x7f5df00a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 -- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00a70c0 msgr2=0x7f5df014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00a70c0 0x7f5df014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 -- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5df00a4580 msgr2=0x7f5df00a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5df00a4580 0x7f5df00a6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 -- 172.21.15.151:0/1250245398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df00a6390 con 0x7f5df006d200 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df4e8d700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011990 tx=0x7f5ddc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc015670 con 0x7f5df006d200 2023-09-30T01:58:48.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc0157d0 con 0x7f5df006d200 2023-09-30T01:58:48.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df006dcd0 con 0x7f5df006d200 2023-09-30T01:58:48.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc015b70 con 0x7f5df006d200 2023-09-30T01:58:48.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df014c240 con 0x7f5df006d200 2023-09-30T01:58:48.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.045+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df00008d0 con 0x7f5df006d200 2023-09-30T01:58:48.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.049+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5ddc013410 con 0x7f5df006d200 2023-09-30T01:58:48.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.049+0000 7f5decff9700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 0x7f5dd4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.049+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5ddc07c880 con 0x7f5df006d200 2023-09-30T01:58:48.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.049+0000 7f5deffff700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 0x7f5dd4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.049+0000 7f5deffff700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 0x7f5dd4062a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5df0138270 tx=0x7f5df00aa7a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.053+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ddc04d060 con 0x7f5df006d200 2023-09-30T01:58:48.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.253+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5df00abdd0 con 0x7f5df006d200 2023-09-30T01:58:48.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.253+0000 7f5decff9700 1 -- 172.21.15.151:0/1250245398 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5ddc050680 con 0x7f5df006d200 2023-09-30T01:58:48.258 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:48.258 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:48.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 msgr2=0x7f5dd4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 0x7f5dd4062a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5df0138270 tx=0x7f5df00aa7a0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 msgr2=0x7f5df00a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011990 tx=0x7f5ddc008040 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 shutdown_connections 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5dd4060580 0x7f5dd4062a30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5df00a70c0 0x7f5df014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5df00a4580 0x7f5df00a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 --2- 172.21.15.151:0/1250245398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5df006d200 0x7f5df00a5e50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 >> 172.21.15.151:0/1250245398 conn(0x7f5df0070510 msgr2=0x7f5df013f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 shutdown_connections 2023-09-30T01:58:48.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:48.257+0000 7f5df5e8f700 1 -- 172.21.15.151:0/1250245398 wait complete. 2023-09-30T01:58:51.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:51.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 Processor -- start 2023-09-30T01:58:51.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 -- start start 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 0x7f30040ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300406c8e0 con 0x7f300406b550 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30040abe70 con 0x7f300406b180 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f300359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30040a9b00 con 0x7f30040ab460 2023-09-30T01:58:51.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38536/0 (socket says 172.21.15.151:38536) 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.381+0000 7f3001d9b700 1 -- 172.21.15.151:0/3305426212 learned_addr learned my addr 172.21.15.151:0/3305426212 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 -- 172.21.15.151:0/3305426212 >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 msgr2=0x7f30040ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300259c700 1 --2- 172.21.15.151:0/3305426212 >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 0x7f30040ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 --2- 172.21.15.151:0/3305426212 >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 0x7f30040ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 -- 172.21.15.151:0/3305426212 >> v1:172.21.15.151:6789/0 conn(0x7f300406b550 legacy=0x7f3004144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 -- 172.21.15.151:0/3305426212 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004141b10 con 0x7f30040ab460 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300259c700 1 --2- 172.21.15.151:0/3305426212 >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 0x7f30040ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 --2- 172.21.15.151:0/3305426212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009f20 tx=0x7f2ff4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34076284d1ba644 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300159a700 1 -- 172.21.15.151:0/3305426212 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400e070 con 0x7f30040ab460 2023-09-30T01:58:51.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300159a700 1 -- 172.21.15.151:0/3305426212 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff400ba80 con 0x7f30040ab460 2023-09-30T01:58:51.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300159a700 1 -- 172.21.15.151:0/3305426212 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40092e0 con 0x7f30040ab460 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- 172.21.15.151:0/3305426212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 msgr2=0x7f30041415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- 172.21.15.151:0/3305426212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009f20 tx=0x7f2ff4009840 comp rx=0 tx=0).stop 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- 172.21.15.151:0/3305426212 shutdown_connections 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- 172.21.15.151:0/3305426212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30040ab460 0x7f30041415d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- 172.21.15.151:0/3305426212 >> v2:172.21.15.172:3300/0 conn(0x7f300406b180 0x7f30040ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- 172.21.15.151:0/3305426212 >> 172.21.15.151:0/3305426212 conn(0x7f3004070510 msgr2=0x7f3004070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- 172.21.15.151:0/3305426212 shutdown_connections 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- 172.21.15.151:0/3305426212 wait complete. 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 Processor -- start 2023-09-30T01:58:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- start start 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30040ab460 0x7f30040a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040adde0 con 0x7f300406b180 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040b0560 con 0x7f30040ab460 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3004140e60 con 0x7f300406b550 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38540/0 (socket says 172.21.15.151:38540) 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f300259c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49132/0 (socket says 172.21.15.151:49132) 2023-09-30T01:58:51.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3001d9b700 1 -- 172.21.15.151:0/2907939799 learned_addr learned my addr 172.21.15.151:0/2907939799 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.385+0000 7f3002d9d700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30040ab460 0x7f30040a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 -- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30040ab460 msgr2=0x7f30040a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30040ab460 0x7f30040a8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 -- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 msgr2=0x7f30041542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 -- 172.21.15.151:0/2907939799 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4009ae0 con 0x7f300406b550 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f300259c700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f3001d9b700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400b6c0 tx=0x7f2ff4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4007040 con 0x7f300406b550 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30040ad970 con 0x7f300406b550 2023-09-30T01:58:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3004154d40 con 0x7f300406b550 2023-09-30T01:58:51.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff40079a0 con 0x7f300406b550 2023-09-30T01:58:51.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4007d60 con 0x7f300406b550 2023-09-30T01:58:51.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2ff4026670 con 0x7f300406b550 2023-09-30T01:58:51.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.389+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30040adf50 con 0x7f300406b550 2023-09-30T01:58:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.393+0000 7f2ff27fc700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 0x7f2fe4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.393+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2ff407e390 con 0x7f300406b550 2023-09-30T01:58:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.393+0000 7f300259c700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 0x7f2fe4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.393+0000 7f300259c700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 0x7f2fe4062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800f440 tx=0x7f2ff8006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.401+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ff404eb70 con 0x7f300406b550 2023-09-30T01:58:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.597+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3004140fa0 con 0x7f300406b550 2023-09-30T01:58:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.597+0000 7f2ff27fc700 1 -- 172.21.15.151:0/2907939799 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2ff4052190 con 0x7f300406b550 2023-09-30T01:58:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:51.601 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 msgr2=0x7f2fe4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 0x7f2fe4062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800f440 tx=0x7f2ff8006f90 comp rx=0 tx=0).stop 2023-09-30T01:58:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 msgr2=0x7f30040a86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400b6c0 tx=0x7f2ff4018000 comp rx=0 tx=0).stop 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 shutdown_connections 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2fe4060550 0x7f2fe4062a00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30040ab460 0x7f30040a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f300406b550 0x7f30040a86b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 --2- 172.21.15.151:0/2907939799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f300406b180 0x7f30041542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 >> 172.21.15.151:0/2907939799 conn(0x7f3004070510 msgr2=0x7f300406e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 shutdown_connections 2023-09-30T01:58:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:51.601+0000 7f300359e700 1 -- 172.21.15.151:0/2907939799 wait complete. 2023-09-30T01:58:54.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 Processor -- start 2023-09-30T01:58:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- start start 2023-09-30T01:58:54.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 0x7fdf2c13d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c13d1e0 con 0x7fdf2c0aa310 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c13e0f0 con 0x7fdf2c070740 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0ac280 con 0x7fdf2c0ab160 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38556/0 (socket says 172.21.15.151:38556) 2023-09-30T01:58:54.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 -- 172.21.15.151:0/1895073210 learned_addr learned my addr 172.21.15.151:0/1895073210 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2ad9d700 1 --2- 172.21.15.151:0/1895073210 >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 0x7fdf2c13d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 -- 172.21.15.151:0/1895073210 >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 msgr2=0x7fdf2c13d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 --2- 172.21.15.151:0/1895073210 >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 0x7fdf2c13d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 -- 172.21.15.151:0/1895073210 >> v1:172.21.15.151:6789/0 conn(0x7fdf2c0aa310 legacy=0x7fdf2c0a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 -- 172.21.15.151:0/1895073210 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c070b10 con 0x7fdf2c0ab160 2023-09-30T01:58:54.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2ad9d700 1 --2- 172.21.15.151:0/1895073210 >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 0x7fdf2c13d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:54.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf2a59c700 1 --2- 172.21.15.151:0/1895073210 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf200099b0 tx=0x7fdf20009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=264d5acd27d3d0d8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf29d9b700 1 -- 172.21.15.151:0/1895073210 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2000b860 con 0x7fdf2c0ab160 2023-09-30T01:58:54.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf29d9b700 1 -- 172.21.15.151:0/1895073210 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2000b9c0 con 0x7fdf2c0ab160 2023-09-30T01:58:54.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf29d9b700 1 -- 172.21.15.151:0/1895073210 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2000bd60 con 0x7fdf2c0ab160 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- 172.21.15.151:0/1895073210 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 msgr2=0x7fdf2c13dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 --2- 172.21.15.151:0/1895073210 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf200099b0 tx=0x7fdf20009250 comp rx=0 tx=0).stop 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- 172.21.15.151:0/1895073210 shutdown_connections 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 --2- 172.21.15.151:0/1895073210 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c0ab160 0x7fdf2c13dbb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 --2- 172.21.15.151:0/1895073210 >> v2:172.21.15.172:3300/0 conn(0x7fdf2c070740 0x7fdf2c13d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- 172.21.15.151:0/1895073210 >> 172.21.15.151:0/1895073210 conn(0x7fdf2c0b0400 msgr2=0x7fdf2c0b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:54.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- 172.21.15.151:0/1895073210 shutdown_connections 2023-09-30T01:58:54.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.725+0000 7fdf30a15700 1 -- 172.21.15.151:0/1895073210 wait complete. 2023-09-30T01:58:54.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 Processor -- start 2023-09-30T01:58:54.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 -- start start 2023-09-30T01:58:54.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf2c0aa310 0x7fdf2c118180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf2c0ab160 0x7fdf2c06bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c119720 con 0x7fdf2c0aa310 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c06b780 con 0x7fdf2c0ab160 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf30a15700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c06d370 con 0x7fdf2c070740 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf2a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf2c0aa310 0x7fdf2c118180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.729+0000 7fdf2ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38568/0 (socket says 172.21.15.151:38568) 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 -- 172.21.15.151:0/243883968 learned_addr learned my addr 172.21.15.151:0/243883968 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2b59e700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf2c0ab160 0x7fdf2c06bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 -- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf2c0ab160 msgr2=0x7fdf2c06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf2c0ab160 0x7fdf2c06bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 -- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf2c0aa310 msgr2=0x7fdf2c118180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf2c0aa310 0x7fdf2c118180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 -- 172.21.15.151:0/243883968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf200094f0 con 0x7fdf2c070740 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf2ad9d700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c860 tx=0x7fdf1c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c011600 con 0x7fdf2c070740 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c011760 con 0x7fdf2c070740 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c1186c0 con 0x7fdf2c070740 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c011b20 con 0x7fdf2c070740 2023-09-30T01:58:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c06c370 con 0x7fdf2c070740 2023-09-30T01:58:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.733+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c0008d0 con 0x7fdf2c070740 2023-09-30T01:58:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.737+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdf1c0193b0 con 0x7fdf2c070740 2023-09-30T01:58:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.737+0000 7fdf12ffd700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 0x7fdf18062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.737+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fdf1c07c540 con 0x7fdf2c070740 2023-09-30T01:58:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.737+0000 7fdf2a59c700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 0x7fdf18062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.737+0000 7fdf2a59c700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 0x7fdf18062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf200099b0 tx=0x7fdf20017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.741+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdf1c04cd20 con 0x7fdf2c070740 2023-09-30T01:58:54.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.933+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdf2c13b260 con 0x7fdf2c070740 2023-09-30T01:58:54.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf12ffd700 1 -- 172.21.15.151:0/243883968 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdf1c050340 con 0x7fdf2c070740 2023-09-30T01:58:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:54.939 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 msgr2=0x7fdf18062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 0x7fdf18062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf200099b0 tx=0x7fdf20017000 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 msgr2=0x7fdf2c06e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c860 tx=0x7fdf1c008040 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 shutdown_connections 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdf18060550 0x7fdf18062a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf2c0ab160 0x7fdf2c06bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdf2c0aa310 0x7fdf2c118180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 --2- 172.21.15.151:0/243883968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdf2c070740 0x7fdf2c06e1e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 >> 172.21.15.151:0/243883968 conn(0x7fdf2c0b0400 msgr2=0x7fdf2c0a9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 shutdown_connections 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:54.937+0000 7fdf30a15700 1 -- 172.21.15.151:0/243883968 wait complete. 2023-09-30T01:58:57.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:58.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 Processor -- start 2023-09-30T01:58:58.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 -- start start 2023-09-30T01:58:58.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3806cbe0 con 0x7f4b380b02a0 2023-09-30T01:58:58.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3806b080 con 0x7f4b3806e2b0 2023-09-30T01:58:58.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ec9d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380b0670 con 0x7f4b380abf60 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38578/0 (socket says 172.21.15.151:38578) 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b3ca39700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4b380b02a0 0x7f4b38141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37056/0 (socket says 172.21.15.151:37056) 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 -- 172.21.15.151:0/1303647674 learned_addr learned my addr 172.21.15.151:0/1303647674 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 -- 172.21.15.151:0/1303647674 >> v1:172.21.15.172:6789/0 conn(0x7f4b3806e2b0 legacy=0x7f4b3806e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 -- 172.21.15.151:0/1303647674 >> v1:172.21.15.151:6789/0 conn(0x7f4b380b02a0 legacy=0x7f4b38141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 -- 172.21.15.151:0/1303647674 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380a9cb0 con 0x7f4b380abf60 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b37fff700 1 --2- 172.21.15.151:0/1303647674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009f50 tx=0x7f4b28009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06ac8e623af8d3a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b377fe700 1 -- 172.21.15.151:0/1303647674 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800e070 con 0x7f4b380abf60 2023-09-30T01:58:58.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b377fe700 1 -- 172.21.15.151:0/1303647674 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800bab0 con 0x7f4b380abf60 2023-09-30T01:58:58.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.065+0000 7f4b377fe700 1 -- 172.21.15.151:0/1303647674 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280092e0 con 0x7f4b380abf60 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/1303647674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 msgr2=0x7f4b3806b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/1303647674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009f50 tx=0x7f4b28009840 comp rx=0 tx=0).stop 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/1303647674 shutdown_connections 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/1303647674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b3806b5a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/1303647674 >> 172.21.15.151:0/1303647674 conn(0x7f4b38070fe0 msgr2=0x7f4b380af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/1303647674 shutdown_connections 2023-09-30T01:58:58.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/1303647674 wait complete. 2023-09-30T01:58:58.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.069+0000 7f4b3ec9d700 1 Processor -- start 2023-09-30T01:58:58.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- start start 2023-09-30T01:58:58.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b3806e2b0 0x7f4b38145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b380b02a0 0x7f4b380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380a88a0 con 0x7f4b380b02a0 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380a8be0 con 0x7f4b3806e2b0 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380a6630 con 0x7f4b380abf60 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38594/0 (socket says 172.21.15.151:38594) 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 -- 172.21.15.151:0/3022574238 learned_addr learned my addr 172.21.15.151:0/3022574238 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3d23a700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b380b02a0 0x7f4b380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ca39700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b3806e2b0 0x7f4b38145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 -- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b3806e2b0 msgr2=0x7f4b38145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b3806e2b0 0x7f4b38145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 -- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b380b02a0 msgr2=0x7f4b380a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b380b02a0 0x7f4b380a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 -- 172.21.15.151:0/3022574238 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009ae0 con 0x7f4b380abf60 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b37fff700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b3806f880 tx=0x7f4b28018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28008d90 con 0x7f4b380abf60 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b28026410 con 0x7f4b380abf60 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b380a5520 con 0x7f4b380abf60 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280267d0 con 0x7f4b380abf60 2023-09-30T01:58:58.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b380a6cb0 con 0x7f4b380abf60 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.073+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b380a95c0 con 0x7f4b380abf60 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.077+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4b2801c410 con 0x7f4b380abf60 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.077+0000 7f4b34ff9700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 0x7f4b24062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.077+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4b2807e5e0 con 0x7f4b380abf60 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.077+0000 7f4b3ca39700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 0x7f4b24062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.077+0000 7f4b3ca39700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 0x7f4b24062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c007fd0 tx=0x7f4b2c007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.081+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4b2804ed20 con 0x7f4b380abf60 2023-09-30T01:58:58.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.277+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4b380a67c0 con 0x7f4b380abf60 2023-09-30T01:58:58.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.277+0000 7f4b34ff9700 1 -- 172.21.15.151:0/3022574238 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4b28052340 con 0x7f4b380abf60 2023-09-30T01:58:58.282 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:58:58.283 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 msgr2=0x7f4b24062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 0x7f4b24062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c007fd0 tx=0x7f4b2c007d70 comp rx=0 tx=0).stop 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 msgr2=0x7f4b380a9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b3806f880 tx=0x7f4b28018000 comp rx=0 tx=0).stop 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 shutdown_connections 2023-09-30T01:58:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4b24060550 0x7f4b24062a00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b380b02a0 0x7f4b380a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b380abf60 0x7f4b380a9080 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 --2- 172.21.15.151:0/3022574238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b3806e2b0 0x7f4b38145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 >> 172.21.15.151:0/3022574238 conn(0x7f4b38070fe0 msgr2=0x7f4b380ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 shutdown_connections 2023-09-30T01:58:58.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:58:58.281+0000 7f4b3ec9d700 1 -- 172.21.15.151:0/3022574238 wait complete. 2023-09-30T01:59:01.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:01.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.405+0000 7f71f610c700 1 Processor -- start 2023-09-30T01:59:01.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.405+0000 7f71f610c700 1 -- start start 2023-09-30T01:59:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 0x7f71f013e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 0x7f71f006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00ac130 con 0x7f71f013ebf0 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00ab550 con 0x7f71f0070740 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f610c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0070b10 con 0x7f71f013f050 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f4909700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 0x7f71f006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f510a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 0x7f71f013e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:40582/0 (socket says 172.21.15.151:40582) 2023-09-30T01:59:01.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 -- 172.21.15.151:0/2252112196 learned_addr learned my addr 172.21.15.151:0/2252112196 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:01.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 -- 172.21.15.151:0/2252112196 >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 msgr2=0x7f71f013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 0x7f71f013e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 -- 172.21.15.151:0/2252112196 >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 msgr2=0x7f71f006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 0x7f71f006d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 -- 172.21.15.151:0/2252112196 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f00ac330 con 0x7f71f013f050 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f510a700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 0x7f71f013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f4909700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 0x7f71f006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71f590b700 1 --2- 172.21.15.151:0/2252112196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009f20 tx=0x7f71e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a460eebf1e370d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.409+0000 7f71e7fff700 1 -- 172.21.15.151:0/2252112196 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800e070 con 0x7f71f013f050 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71e7fff700 1 -- 172.21.15.151:0/2252112196 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e800b930 con 0x7f71f013f050 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71e7fff700 1 -- 172.21.15.151:0/2252112196 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e80092e0 con 0x7f71f013f050 2023-09-30T01:59:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- 172.21.15.151:0/2252112196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 msgr2=0x7f71f006deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- 172.21.15.151:0/2252112196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009f20 tx=0x7f71e8009840 comp rx=0 tx=0).stop 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- 172.21.15.151:0/2252112196 shutdown_connections 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- 172.21.15.151:0/2252112196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f006deb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.151:3300/0 conn(0x7f71f013ebf0 0x7f71f006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- 172.21.15.151:0/2252112196 >> v2:172.21.15.172:3300/0 conn(0x7f71f0070740 0x7f71f013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- 172.21.15.151:0/2252112196 >> 172.21.15.151:0/2252112196 conn(0x7f71f00b03f0 msgr2=0x7f71f00b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- 172.21.15.151:0/2252112196 shutdown_connections 2023-09-30T01:59:01.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- 172.21.15.151:0/2252112196 wait complete. 2023-09-30T01:59:01.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 Processor -- start 2023-09-30T01:59:01.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- start start 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 0x7f71f00aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 0x7f71f01381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013b9d0 con 0x7f71f013ebf0 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013bcb0 con 0x7f71f0070740 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f610c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013a490 con 0x7f71f013f050 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f590b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 0x7f71f01381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f510a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 0x7f71f00aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f590b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:40588/0 (socket says 172.21.15.151:40588) 2023-09-30T01:59:01.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f590b700 1 -- 172.21.15.151:0/2600877266 learned_addr learned my addr 172.21.15.151:0/2600877266 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 -- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 msgr2=0x7f71f014fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 -- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 msgr2=0x7f71f00aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 0x7f71f00aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 -- 172.21.15.151:0/2600877266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec009420 con 0x7f71f013ebf0 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f590b700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f510a700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 0x7f71f00aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.413+0000 7f71f4909700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 0x7f71f01381d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f71e000ec80 tx=0x7f71e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000c850 con 0x7f71f013ebf0 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000c9b0 con 0x7f71f013ebf0 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e0009420 con 0x7f71f013ebf0 2023-09-30T01:59:01.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000cd70 con 0x7f71f013ebf0 2023-09-30T01:59:01.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8009ae0 con 0x7f71f013ebf0 2023-09-30T01:59:01.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f00008d0 con 0x7f71f013ebf0 2023-09-30T01:59:01.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f71e0013680 con 0x7f71f013ebf0 2023-09-30T01:59:01.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 0x7f71d4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f71e007c870 con 0x7f71f013ebf0 2023-09-30T01:59:01.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.417+0000 7f71f510a700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 0x7f71d4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.421+0000 7f71f510a700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 0x7f71d4062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71ec011600 tx=0x7f71ec0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.426 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.425+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f71e004cfb0 con 0x7f71f013ebf0 2023-09-30T01:59:01.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.617+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f71f00ac610 con 0x7f71f013ebf0 2023-09-30T01:59:01.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.617+0000 7f71e57fa700 1 -- 172.21.15.151:0/2600877266 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f71e00505d0 con 0x7f71f013ebf0 2023-09-30T01:59:01.621 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:01.621 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:01.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 msgr2=0x7f71d4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 0x7f71d4062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71ec011600 tx=0x7f71ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 msgr2=0x7f71f01381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 0x7f71f01381d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f71e000ec80 tx=0x7f71e00091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 shutdown_connections 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f71d4060550 0x7f71d4062a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71f013f050 0x7f71f014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71f013ebf0 0x7f71f01381d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 --2- 172.21.15.151:0/2600877266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71f0070740 0x7f71f00aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 >> 172.21.15.151:0/2600877266 conn(0x7f71f00b03f0 msgr2=0x7f71f00ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 shutdown_connections 2023-09-30T01:59:01.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:01.621+0000 7f71f610c700 1 -- 172.21.15.151:0/2600877266 wait complete. 2023-09-30T01:59:04.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:04.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 Processor -- start 2023-09-30T01:59:04.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 -- start start 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 0x7f9f6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 0x7f9f60141040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600b0d30 con 0x7f9f6006b3d0 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600ae490 con 0x7f9f600b0960 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6006c690 con 0x7f9f60140c10 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5effd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 0x7f9f6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 0x7f9f60141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5e7fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9f6006b3d0 0x7f9f6006d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41390/0 (socket says 172.21.15.151:41390) 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5e7fc700 1 -- 172.21.15.151:0/4209303218 learned_addr learned my addr 172.21.15.151:0/4209303218 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 -- 172.21.15.151:0/4209303218 >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 msgr2=0x7f9f6006d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 --2- 172.21.15.151:0/4209303218 >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 0x7f9f6006d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 -- 172.21.15.151:0/4209303218 >> v1:172.21.15.151:6789/0 conn(0x7f9f6006b3d0 legacy=0x7f9f6006d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 -- 172.21.15.151:0/4209303218 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6006c970 con 0x7f9f60140c10 2023-09-30T01:59:04.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5effd700 1 --2- 172.21.15.151:0/4209303218 >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 0x7f9f6006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:04.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5f7fe700 1 --2- 172.21.15.151:0/4209303218 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 0x7f9f60141040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9f500098e0 tx=0x7f9f500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f58353b32bceda5c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:04.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5dffb700 1 -- 172.21.15.151:0/4209303218 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000b640 con 0x7f9f60140c10 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5dffb700 1 -- 172.21.15.151:0/4209303218 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5000b7a0 con 0x7f9f60140c10 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f5dffb700 1 -- 172.21.15.151:0/4209303218 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000bb40 con 0x7f9f60140c10 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 -- 172.21.15.151:0/4209303218 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 msgr2=0x7f9f60141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.745+0000 7f9f65324700 1 --2- 172.21.15.151:0/4209303218 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 0x7f9f60141040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9f500098e0 tx=0x7f9f500091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 -- 172.21.15.151:0/4209303218 shutdown_connections 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 --2- 172.21.15.151:0/4209303218 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f60140c10 0x7f9f60141040 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 --2- 172.21.15.151:0/4209303218 >> v2:172.21.15.151:3300/0 conn(0x7f9f600b0960 0x7f9f6006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 -- 172.21.15.151:0/4209303218 >> 172.21.15.151:0/4209303218 conn(0x7f9f60070510 msgr2=0x7f9f60070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 -- 172.21.15.151:0/4209303218 shutdown_connections 2023-09-30T01:59:04.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 -- 172.21.15.151:0/4209303218 wait complete. 2023-09-30T01:59:04.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 Processor -- start 2023-09-30T01:59:04.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.749+0000 7f9f65324700 1 -- start start 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f6006b3d0 0x7f9f600ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 0x7f9f6011dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6011d710 con 0x7f9f6006b3d0 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6011bf00 con 0x7f9f60140c10 2023-09-30T01:59:04.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6011e0e0 con 0x7f9f600b0960 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f6006b3d0 0x7f9f600ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:40610/0 (socket says 172.21.15.151:40610) 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 -- 172.21.15.151:0/448045857 learned_addr learned my addr 172.21.15.151:0/448045857 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5f7fe700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 0x7f9f6011dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 -- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 msgr2=0x7f9f6011dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 0x7f9f6011dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 -- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f6006b3d0 msgr2=0x7f9f600ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f6006b3d0 0x7f9f600ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 -- 172.21.15.151:0/448045857 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58009420 con 0x7f9f600b0960 2023-09-30T01:59:04.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5f7fe700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 0x7f9f6011dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f5e7fc700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800cc70 tx=0x7f9f48001a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4800a940 con 0x7f9f600b0960 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4800aac0 con 0x7f9f600b0960 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f50009420 con 0x7f9f600b0960 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4800ae80 con 0x7f9f600b0960 2023-09-30T01:59:04.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.753+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6011cee0 con 0x7f9f600b0960 2023-09-30T01:59:04.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9f48007300 con 0x7f9f600b0960 2023-09-30T01:59:04.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6011d880 con 0x7f9f600b0960 2023-09-30T01:59:04.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f56ffd700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 0x7f9f40062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:04.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9f4807c920 con 0x7f9f600b0960 2023-09-30T01:59:04.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f5effd700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 0x7f9f40062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:04.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.757+0000 7f9f5effd700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 0x7f9f40062a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009f40 tx=0x7f9f580091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:04.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.765+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9f4804d180 con 0x7f9f600b0960 2023-09-30T01:59:04.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.957+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9f600670a0 con 0x7f9f600b0960 2023-09-30T01:59:04.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.957+0000 7f9f56ffd700 1 -- 172.21.15.151:0/448045857 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9f480507a0 con 0x7f9f600b0960 2023-09-30T01:59:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:04.961 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:04.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 msgr2=0x7f9f40062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 0x7f9f40062a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009f40 tx=0x7f9f580091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 msgr2=0x7f9f6011b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:04.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800cc70 tx=0x7f9f48001a30 comp rx=0 tx=0).stop 2023-09-30T01:59:04.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 shutdown_connections 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9f40060550 0x7f9f40062a00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9f60140c10 0x7f9f6011dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f600b0960 0x7f9f6011b9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 --2- 172.21.15.151:0/448045857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f6006b3d0 0x7f9f600ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 >> 172.21.15.151:0/448045857 conn(0x7f9f60070510 msgr2=0x7f9f6013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 shutdown_connections 2023-09-30T01:59:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:04.961+0000 7f9f65324700 1 -- 172.21.15.151:0/448045857 wait complete. 2023-09-30T01:59:07.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:08.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 Processor -- start 2023-09-30T01:59:08.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 -- start start 2023-09-30T01:59:08.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 0x7f0b3406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b3406eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b340adf40 con 0x7f0b3406b770 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3406b590 con 0x7f0b340b44f0 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3406d020 con 0x7f0b3406ea60 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b31d9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0b3406b770 0x7f0b34140790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41402/0 (socket says 172.21.15.151:41402) 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b31d9b700 1 -- 172.21.15.151:0/622985327 learned_addr learned my addr 172.21.15.151:0/622985327 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b32d9d700 1 --2- 172.21.15.151:0/622985327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b3406eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3259c700 1 --2- 172.21.15.151:0/622985327 >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 0x7f0b3406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3159a700 1 -- 172.21.15.151:0/622985327 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354755086 0 0) 0x7f0b340adf40 con 0x7f0b3406b770 2023-09-30T01:59:08.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3159a700 1 -- 172.21.15.151:0/622985327 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b20003570 con 0x7f0b3406b770 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b32d9d700 1 -- 172.21.15.151:0/622985327 >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 msgr2=0x7f0b3406b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b32d9d700 1 --2- 172.21.15.151:0/622985327 >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 0x7f0b3406b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b32d9d700 1 -- 172.21.15.151:0/622985327 >> v1:172.21.15.151:6789/0 conn(0x7f0b3406b770 legacy=0x7f0b34140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b32d9d700 1 -- 172.21.15.151:0/622985327 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b340b48c0 con 0x7f0b3406ea60 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.085+0000 7f0b3259c700 1 --2- 172.21.15.151:0/622985327 >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 0x7f0b3406b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b32d9d700 1 --2- 172.21.15.151:0/622985327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b3406eec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0098e0 tx=0x7f0b2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bfe8ea61a963710 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3159a700 1 -- 172.21.15.151:0/622985327 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00b640 con 0x7f0b3406ea60 2023-09-30T01:59:08.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3159a700 1 -- 172.21.15.151:0/622985327 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c00b7a0 con 0x7f0b3406ea60 2023-09-30T01:59:08.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- 172.21.15.151:0/622985327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 msgr2=0x7f0b3406eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- 172.21.15.151:0/622985327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b3406eec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0098e0 tx=0x7f0b2c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- 172.21.15.151:0/622985327 shutdown_connections 2023-09-30T01:59:08.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- 172.21.15.151:0/622985327 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b3406eec0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- 172.21.15.151:0/622985327 >> v2:172.21.15.172:3300/0 conn(0x7f0b340b44f0 0x7f0b3406b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- 172.21.15.151:0/622985327 >> 172.21.15.151:0/622985327 conn(0x7f0b34070fe0 msgr2=0x7f0b340b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:08.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- 172.21.15.151:0/622985327 shutdown_connections 2023-09-30T01:59:08.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- 172.21.15.151:0/622985327 wait complete. 2023-09-30T01:59:08.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 Processor -- start 2023-09-30T01:59:08.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- start start 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3406b770 0x7f0b34150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b340b44f0 0x7f0b340a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34150e80 con 0x7f0b340b44f0 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341510d0 con 0x7f0b3406b770 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.089+0000 7f0b3359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b340ab5f0 con 0x7f0b3406ea60 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b32d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b340b44f0 0x7f0b340a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:40620/0 (socket says 172.21.15.151:40620) 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 -- 172.21.15.151:0/4144998278 learned_addr learned my addr 172.21.15.151:0/4144998278 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b3259c700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3406b770 0x7f0b34150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 -- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3406b770 msgr2=0x7f0b34150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3406b770 0x7f0b34150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 -- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b340b44f0 msgr2=0x7f0b340a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b340b44f0 0x7f0b340a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 -- 172.21.15.151:0/4144998278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c009420 con 0x7f0b3406ea60 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b31d9b700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400cd30 tx=0x7f0b24001b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400ce90 con 0x7f0b3406ea60 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2400aac0 con 0x7f0b3406ea60 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b340ab8d0 con 0x7f0b3406ea60 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b340a9d10 con 0x7f0b3406ea60 2023-09-30T01:59:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400aea0 con 0x7f0b3406ea60 2023-09-30T01:59:08.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b340a9270 con 0x7f0b3406ea60 2023-09-30T01:59:08.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.093+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0b24007300 con 0x7f0b3406ea60 2023-09-30T01:59:08.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.097+0000 7f0b2a7fc700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 0x7f0b14062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.097+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0b2407ca30 con 0x7f0b3406ea60 2023-09-30T01:59:08.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.097+0000 7f0b3259c700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 0x7f0b14062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.097+0000 7f0b3259c700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 0x7f0b14062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c006f00 tx=0x7f0b1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.105+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0b340a9270 con 0x7f0b3406ea60 2023-09-30T01:59:08.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.297+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0b340a9270 con 0x7f0b3406ea60 2023-09-30T01:59:08.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b2a7fc700 1 -- 172.21.15.151:0/4144998278 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0b340a9270 con 0x7f0b3406ea60 2023-09-30T01:59:08.303 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:08.303 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 msgr2=0x7f0b14062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 0x7f0b14062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c006f00 tx=0x7f0b1c008040 comp rx=0 tx=0).stop 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 msgr2=0x7f0b34150940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400cd30 tx=0x7f0b24001b10 comp rx=0 tx=0).stop 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 shutdown_connections 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0b14060550 0x7f0b14062a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b340b44f0 0x7f0b340a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b3406ea60 0x7f0b34150940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 --2- 172.21.15.151:0/4144998278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b3406b770 0x7f0b34150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 >> 172.21.15.151:0/4144998278 conn(0x7f0b34070fe0 msgr2=0x7f0b3406e020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 shutdown_connections 2023-09-30T01:59:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:08.301+0000 7f0b3359e700 1 -- 172.21.15.151:0/4144998278 wait complete. 2023-09-30T01:59:11.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:11.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 Processor -- start 2023-09-30T01:59:11.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 -- start start 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 0x7f316806d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316806c4b0 con 0x7f316806db80 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316806c720 con 0x7f31680b3f60 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3167fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316806ca00 con 0x7f31680b3b90 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3166ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3166ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:37192/0 (socket says 172.21.15.151:37192) 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f3166ffd700 1 -- 172.21.15.151:0/3468138496 learned_addr learned my addr 172.21.15.151:0/3468138496 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:11.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.425+0000 7f31667fc700 1 --2- 172.21.15.151:0/3468138496 >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 0x7f316806d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 -- 172.21.15.151:0/3468138496 >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 msgr2=0x7f316806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 --2- 172.21.15.151:0/3468138496 >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 0x7f316806d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 -- 172.21.15.151:0/3468138496 >> v1:172.21.15.172:6789/0 conn(0x7f316806db80 legacy=0x7f31680ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 -- 172.21.15.151:0/3468138496 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316806b810 con 0x7f31680b3b90 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f31667fc700 1 --2- 172.21.15.151:0/3468138496 >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 0x7f316806d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:11.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 --2- 172.21.15.151:0/3468138496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f315c0099b0 tx=0x7f315c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73fbd090e2863c21 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3165ffb700 1 -- 172.21.15.151:0/3468138496 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c00b860 con 0x7f31680b3b90 2023-09-30T01:59:11.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3165ffb700 1 -- 172.21.15.151:0/3468138496 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c00b9c0 con 0x7f31680b3b90 2023-09-30T01:59:11.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3165ffb700 1 -- 172.21.15.151:0/3468138496 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c00bd80 con 0x7f31680b3b90 2023-09-30T01:59:11.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- 172.21.15.151:0/3468138496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 msgr2=0x7f316806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- 172.21.15.151:0/3468138496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f315c0099b0 tx=0x7f315c009250 comp rx=0 tx=0).stop 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- 172.21.15.151:0/3468138496 shutdown_connections 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- 172.21.15.151:0/3468138496 >> v2:172.21.15.172:3300/0 conn(0x7f31680b3f60 0x7f316806d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- 172.21.15.151:0/3468138496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316806d100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- 172.21.15.151:0/3468138496 >> 172.21.15.151:0/3468138496 conn(0x7f3168063990 msgr2=0x7f3168070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- 172.21.15.151:0/3468138496 shutdown_connections 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- 172.21.15.151:0/3468138496 wait complete. 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 Processor -- start 2023-09-30T01:59:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- start start 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f316806db80 0x7f316813e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680b3f60 0x7f3168150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680aad80 con 0x7f31680b3f60 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680ac5e0 con 0x7f316806db80 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3167fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680ab930 con 0x7f31680b3b90 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f31667fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f31667fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:37208/0 (socket says 172.21.15.151:37208) 2023-09-30T01:59:11.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f31667fc700 1 -- 172.21.15.151:0/1427339337 learned_addr learned my addr 172.21.15.151:0/1427339337 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f31677fe700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680b3f60 0x7f3168150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.429+0000 7f3166ffd700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f316806db80 0x7f316813e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 -- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f316806db80 msgr2=0x7f316813e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f316806db80 0x7f316813e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 -- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680b3f60 msgr2=0x7f3168150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680b3f60 0x7f3168150410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 -- 172.21.15.151:0/1427339337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0094f0 con 0x7f31680b3b90 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f31667fc700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f315800e8d0 tx=0x7f3158008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158013410 con 0x7f31680b3b90 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158013570 con 0x7f31680b3b90 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3168150950 con 0x7f31680b3b90 2023-09-30T01:59:11.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158013910 con 0x7f31680b3b90 2023-09-30T01:59:11.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3168150e20 con 0x7f31680b3b90 2023-09-30T01:59:11.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680ac750 con 0x7f31680b3b90 2023-09-30T01:59:11.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3158013d00 con 0x7f31680b3b90 2023-09-30T01:59:11.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.433+0000 7f314effd700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 0x7f3154062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.437+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f315807c440 con 0x7f31680b3b90 2023-09-30T01:59:11.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.437+0000 7f3166ffd700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 0x7f3154062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.437+0000 7f3166ffd700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 0x7f3154062a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315c0099b0 tx=0x7f315c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.441+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f315804cb80 con 0x7f31680b3b90 2023-09-30T01:59:11.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31680abac0 con 0x7f31680b3b90 2023-09-30T01:59:11.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f314effd700 1 -- 172.21.15.151:0/1427339337 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31580501a0 con 0x7f31680b3b90 2023-09-30T01:59:11.640 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:11.640 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 msgr2=0x7f3154062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 0x7f3154062a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315c0099b0 tx=0x7f315c017000 comp rx=0 tx=0).stop 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 msgr2=0x7f316811b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f315800e8d0 tx=0x7f3158008040 comp rx=0 tx=0).stop 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 shutdown_connections 2023-09-30T01:59:11.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3154060550 0x7f3154062a00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680b3f60 0x7f3168150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31680b3b90 0x7f316811b870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 --2- 172.21.15.151:0/1427339337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f316806db80 0x7f316813e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 >> 172.21.15.151:0/1427339337 conn(0x7f3168063990 msgr2=0x7f3168143010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 shutdown_connections 2023-09-30T01:59:11.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:11.637+0000 7f3167fff700 1 -- 172.21.15.151:0/1427339337 wait complete. 2023-09-30T01:59:14.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:14.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.765+0000 7efe3d4e3700 1 Processor -- start 2023-09-30T01:59:14.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.765+0000 7efe3d4e3700 1 -- start start 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 0x7efe380ad550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efe38149390 0x7efe3814b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3806c850 con 0x7efe380675a0 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe380acf30 con 0x7efe38149390 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe380ae180 con 0x7efe380abf20 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efe38149390 0x7efe3814b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efe38149390 0x7efe3814b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39390/0 (socket says 172.21.15.151:39390) 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 -- 172.21.15.151:0/3812191959 learned_addr learned my addr 172.21.15.151:0/3812191959 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe377fe700 1 --2- 172.21.15.151:0/3812191959 >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 0x7efe380ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 -- 172.21.15.151:0/3812191959 >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 msgr2=0x7efe380ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 --2- 172.21.15.151:0/3812191959 >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 0x7efe380ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 -- 172.21.15.151:0/3812191959 >> v1:172.21.15.172:6789/0 conn(0x7efe380675a0 legacy=0x7efe380ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 -- 172.21.15.151:0/3812191959 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe38067970 con 0x7efe38149390 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe377fe700 1 --2- 172.21.15.151:0/3812191959 >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 0x7efe380ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3cce2700 1 --2- 172.21.15.151:0/3812191959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe3814b770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efe38071460 tx=0x7efe30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef10782c08450813 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe36ffd700 1 -- 172.21.15.151:0/3812191959 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3000e070 con 0x7efe38149390 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe36ffd700 1 -- 172.21.15.151:0/3812191959 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3000ba40 con 0x7efe38149390 2023-09-30T01:59:14.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe36ffd700 1 -- 172.21.15.151:0/3812191959 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe300092e0 con 0x7efe38149390 2023-09-30T01:59:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- 172.21.15.151:0/3812191959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 msgr2=0x7efe3814b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/3812191959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe3814b770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efe38071460 tx=0x7efe30009840 comp rx=0 tx=0).stop 2023-09-30T01:59:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- 172.21.15.151:0/3812191959 shutdown_connections 2023-09-30T01:59:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/3812191959 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe3814b770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/3812191959 >> v2:172.21.15.172:3300/0 conn(0x7efe380abf20 0x7efe380ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- 172.21.15.151:0/3812191959 >> 172.21.15.151:0/3812191959 conn(0x7efe3806ae50 msgr2=0x7efe38070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:14.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- 172.21.15.151:0/3812191959 shutdown_connections 2023-09-30T01:59:14.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.769+0000 7efe3d4e3700 1 -- 172.21.15.151:0/3812191959 wait complete. 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 Processor -- start 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- start start 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe380675a0 0x7efe38145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe38154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe380a6980 con 0x7efe38149390 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe380a8800 con 0x7efe380675a0 2023-09-30T01:59:14.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe380a65f0 con 0x7efe380abf20 2023-09-30T01:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3cce2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe38154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:37210/0 (socket says 172.21.15.151:37210) 2023-09-30T01:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 -- 172.21.15.151:0/2443517190 learned_addr learned my addr 172.21.15.151:0/2443517190 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe37fff700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe380675a0 0x7efe38145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 -- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe380675a0 msgr2=0x7efe38145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe380675a0 0x7efe38145410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 -- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 msgr2=0x7efe38154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe38154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 -- 172.21.15.151:0/2443517190 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30009ae0 con 0x7efe380abf20 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3cce2700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe38154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe377fe700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e8d0 tx=0x7efe28008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:14.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe28013410 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe28013570 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe381460b0 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe28013910 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.773+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe380a60f0 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efe28013d00 con 0x7efe380abf20 2023-09-30T01:59:14.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe380a6af0 con 0x7efe380abf20 2023-09-30T01:59:14.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe1ffff700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 0x7efe24062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:14.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efe2807c700 con 0x7efe380abf20 2023-09-30T01:59:14.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe37fff700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 0x7efe24062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:14.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.777+0000 7efe37fff700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 0x7efe24062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efe3806c9e0 tx=0x7efe2c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:14.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.785+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efe280508d0 con 0x7efe380abf20 2023-09-30T01:59:14.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.981+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efe380a6730 con 0x7efe380abf20 2023-09-30T01:59:14.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe1ffff700 1 -- 172.21.15.151:0/2443517190 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efe28050460 con 0x7efe380abf20 2023-09-30T01:59:14.987 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:14.987 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:14.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 msgr2=0x7efe24062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 0x7efe24062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efe3806c9e0 tx=0x7efe2c007ec0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 msgr2=0x7efe38154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:14.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e8d0 tx=0x7efe28008040 comp rx=0 tx=0).stop 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 shutdown_connections 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efe24060550 0x7efe24062a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe38149390 0x7efe38154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe380abf20 0x7efe38154600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 --2- 172.21.15.151:0/2443517190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efe380675a0 0x7efe38145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 >> 172.21.15.151:0/2443517190 conn(0x7efe3806ae50 msgr2=0x7efe38143a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 shutdown_connections 2023-09-30T01:59:14.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:14.985+0000 7efe3d4e3700 1 -- 172.21.15.151:0/2443517190 wait complete. 2023-09-30T01:59:18.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:18.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.109+0000 7f05e7fff700 1 Processor -- start 2023-09-30T01:59:18.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.109+0000 7f05e7fff700 1 -- start start 2023-09-30T01:59:18.116 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f05e806e330 0x7f05e806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e806e700 con 0x7f05e806c490 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e8137d10 con 0x7f05e806e330 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e806c940 con 0x7f05e8137870 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e6ffd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f05e806e330 0x7f05e806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:37224/0 (socket says 172.21.15.151:37224) 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 -- 172.21.15.151:0/315573132 learned_addr learned my addr 172.21.15.151:0/315573132 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05dffff700 1 --1- 172.21.15.151:0/315573132 >> v1:172.21.15.151:6789/0 conn(0x7f05e806c490 0x7f05e8137160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44230/0 (socket says 172.21.15.151:44230) 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e67fc700 1 -- 172.21.15.151:0/315573132 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22388465 0 0) 0x7f05e806e700 con 0x7f05e806c490 2023-09-30T01:59:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e67fc700 1 -- 172.21.15.151:0/315573132 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d4003570 con 0x7f05e806c490 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 -- 172.21.15.151:0/315573132 >> v2:172.21.15.151:3300/0 conn(0x7f05e806e330 msgr2=0x7f05e806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 --2- 172.21.15.151:0/315573132 >> v2:172.21.15.151:3300/0 conn(0x7f05e806e330 0x7f05e806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 -- 172.21.15.151:0/315573132 >> v1:172.21.15.151:6789/0 conn(0x7f05e806c490 legacy=0x7f05e8137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 -- 172.21.15.151:0/315573132 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e80a7920 con 0x7f05e8137870 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e77fe700 1 --2- 172.21.15.151:0/315573132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009f50 tx=0x7f05e0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d551a2316f9ba21c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e67fc700 1 -- 172.21.15.151:0/315573132 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e000e070 con 0x7f05e8137870 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e67fc700 1 -- 172.21.15.151:0/315573132 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e000b9c0 con 0x7f05e8137870 2023-09-30T01:59:18.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e67fc700 1 -- 172.21.15.151:0/315573132 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e00092e0 con 0x7f05e8137870 2023-09-30T01:59:18.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- 172.21.15.151:0/315573132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 msgr2=0x7f05e813a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 --2- 172.21.15.151:0/315573132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009f50 tx=0x7f05e0009840 comp rx=0 tx=0).stop 2023-09-30T01:59:18.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- 172.21.15.151:0/315573132 shutdown_connections 2023-09-30T01:59:18.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 --2- 172.21.15.151:0/315573132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e8137870 0x7f05e813a220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 --2- 172.21.15.151:0/315573132 >> v2:172.21.15.151:3300/0 conn(0x7f05e806e330 0x7f05e806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.113+0000 7f05e7fff700 1 -- 172.21.15.151:0/315573132 >> 172.21.15.151:0/315573132 conn(0x7f05e806ae50 msgr2=0x7f05e8070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:18.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- 172.21.15.151:0/315573132 shutdown_connections 2023-09-30T01:59:18.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- 172.21.15.151:0/315573132 wait complete. 2023-09-30T01:59:18.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 Processor -- start 2023-09-30T01:59:18.121 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- start start 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 0x7f05e80a8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05e8137870 0x7f05e80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e8150a40 con 0x7f05e806c490 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80a8220 con 0x7f05e8137870 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e7fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80a9c40 con 0x7f05e806e330 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05dffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 0x7f05e80a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39408/0 (socket says 172.21.15.151:39408) 2023-09-30T01:59:18.122 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 -- 172.21.15.151:0/3026821182 learned_addr learned my addr 172.21.15.151:0/3026821182 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:18.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e77fe700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05e8137870 0x7f05e80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 -- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 msgr2=0x7f05e80a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 0x7f05e80a8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 -- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05e8137870 msgr2=0x7f05e80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05e8137870 0x7f05e80a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05e6ffd700 1 -- 172.21.15.151:0/3026821182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0009ae0 con 0x7f05e806c490 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.117+0000 7f05dffff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 0x7f05e80a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05e6ffd700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05d800d8c0 tx=0x7f05d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d80125f0 con 0x7f05e806c490 2023-09-30T01:59:18.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d8012750 con 0x7f05e806c490 2023-09-30T01:59:18.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e80a9eb0 con 0x7f05e806c490 2023-09-30T01:59:18.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8012af0 con 0x7f05e806c490 2023-09-30T01:59:18.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e814fb40 con 0x7f05e806c490 2023-09-30T01:59:18.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e80008d0 con 0x7f05e806c490 2023-09-30T01:59:18.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.121+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f05d801e460 con 0x7f05e806c490 2023-09-30T01:59:18.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.125+0000 7f05deffd700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 0x7f05c8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.125+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f05d807cc30 con 0x7f05e806c490 2023-09-30T01:59:18.127 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.125+0000 7f05dffff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 0x7f05c8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.128 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.125+0000 7f05dffff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 0x7f05c8062a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05e813ae20 tx=0x7f05d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.129+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f05d804d370 con 0x7f05e806c490 2023-09-30T01:59:18.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f05e80ad930 con 0x7f05e806c490 2023-09-30T01:59:18.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05deffd700 1 -- 172.21.15.151:0/3026821182 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f05d8050990 con 0x7f05e806c490 2023-09-30T01:59:18.331 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:18.331 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 msgr2=0x7f05c8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 0x7f05c8062a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f05e813ae20 tx=0x7f05d0009040 comp rx=0 tx=0).stop 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 msgr2=0x7f05e8136570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05d800d8c0 tx=0x7f05d8008040 comp rx=0 tx=0).stop 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 shutdown_connections 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f05c8060580 0x7f05c8062a30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05e8137870 0x7f05e80a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f05e806e330 0x7f05e80a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 --2- 172.21.15.151:0/3026821182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f05e806c490 0x7f05e8136570 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 >> 172.21.15.151:0/3026821182 conn(0x7f05e806ae50 msgr2=0x7f05e80ac6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:18.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 shutdown_connections 2023-09-30T01:59:18.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:18.329+0000 7f05e7fff700 1 -- 172.21.15.151:0/3026821182 wait complete. 2023-09-30T01:59:21.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:21.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 Processor -- start 2023-09-30T01:59:21.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 -- start start 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 0x7f6d5013ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500ab3e0 con 0x7f6d50070740 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d50070b10 con 0x7f6d5013a050 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d56b83700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500ab630 con 0x7f6d500aae90 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55b81700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6d50070740 0x7f6d5013d6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58438/0 (socket says 172.21.15.151:58438) 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55380700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 0x7f6d5013ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55b81700 1 -- 172.21.15.151:0/3359237555 learned_addr learned my addr 172.21.15.151:0/3359237555 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d54b7f700 1 -- 172.21.15.151:0/3359237555 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189060244 0 0) 0x7f6d500ab3e0 con 0x7f6d50070740 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d54b7f700 1 -- 172.21.15.151:0/3359237555 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38003570 con 0x7f6d50070740 2023-09-30T01:59:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55380700 1 -- 172.21.15.151:0/3359237555 >> v1:172.21.15.172:6789/0 conn(0x7f6d5013a050 legacy=0x7f6d5013a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55380700 1 -- 172.21.15.151:0/3359237555 >> v1:172.21.15.151:6789/0 conn(0x7f6d50070740 legacy=0x7f6d5013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55380700 1 -- 172.21.15.151:0/3359237555 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500a8850 con 0x7f6d500aae90 2023-09-30T01:59:21.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d55380700 1 --2- 172.21.15.151:0/3359237555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 0x7f6d5013ddd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009910 tx=0x7f6d4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7303ad9920e73772 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:21.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d54b7f700 1 -- 172.21.15.151:0/3359237555 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c00b680 con 0x7f6d500aae90 2023-09-30T01:59:21.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d54b7f700 1 -- 172.21.15.151:0/3359237555 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c00b7e0 con 0x7f6d500aae90 2023-09-30T01:59:21.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.457+0000 7f6d54b7f700 1 -- 172.21.15.151:0/3359237555 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c00bba0 con 0x7f6d500aae90 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- 172.21.15.151:0/3359237555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 msgr2=0x7f6d5013ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 --2- 172.21.15.151:0/3359237555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 0x7f6d5013ddd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009910 tx=0x7f6d4c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- 172.21.15.151:0/3359237555 shutdown_connections 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 --2- 172.21.15.151:0/3359237555 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d500aae90 0x7f6d5013ddd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- 172.21.15.151:0/3359237555 >> 172.21.15.151:0/3359237555 conn(0x7f6d500b03f0 msgr2=0x7f6d500b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- 172.21.15.151:0/3359237555 shutdown_connections 2023-09-30T01:59:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- 172.21.15.151:0/3359237555 wait complete. 2023-09-30T01:59:21.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 Processor -- start 2023-09-30T01:59:21.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- start start 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d500aae90 0x7f6d5014d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d5013a050 0x7f6d5014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5013fbb0 con 0x7f6d5013a050 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5006e6e0 con 0x7f6d500aae90 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56b83700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50117db0 con 0x7f6d50070740 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d55b81700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:21.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d55b81700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:57256/0 (socket says 172.21.15.151:57256) 2023-09-30T01:59:21.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d55b81700 1 -- 172.21.15.151:0/4264355984 learned_addr learned my addr 172.21.15.151:0/4264355984 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:21.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d56382700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d5013a050 0x7f6d5014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:21.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.461+0000 7f6d55380700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d500aae90 0x7f6d5014d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 -- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d500aae90 msgr2=0x7f6d5014d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d500aae90 0x7f6d5014d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 -- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d5013a050 msgr2=0x7f6d5014d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d5013a050 0x7f6d5014d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 -- 172.21.15.151:0/4264355984 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c009420 con 0x7f6d50070740 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d55b81700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000cd60 tx=0x7f6d40001b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000aa80 con 0x7f6d50070740 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4000ac00 con 0x7f6d50070740 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d400070f0 con 0x7f6d50070740 2023-09-30T01:59:21.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5014db90 con 0x7f6d50070740 2023-09-30T01:59:21.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5006d5d0 con 0x7f6d50070740 2023-09-30T01:59:21.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.465+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d500008d0 con 0x7f6d50070740 2023-09-30T01:59:21.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.469+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6d400074e0 con 0x7f6d50070740 2023-09-30T01:59:21.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.469+0000 7f6d45ffb700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 0x7f6d3c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:21.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.469+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6d4007c9e0 con 0x7f6d50070740 2023-09-30T01:59:21.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.469+0000 7f6d55380700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 0x7f6d3c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:21.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.469+0000 7f6d55380700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 0x7f6d3c062a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0093f0 tx=0x7f6d4c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:21.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.477+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6d4004d240 con 0x7f6d50070740 2023-09-30T01:59:21.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6d50117ef0 con 0x7f6d50070740 2023-09-30T01:59:21.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d45ffb700 1 -- 172.21.15.151:0/4264355984 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6d50117ef0 con 0x7f6d50070740 2023-09-30T01:59:21.672 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:21.672 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:21.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 msgr2=0x7f6d3c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 0x7f6d3c062a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0093f0 tx=0x7f6d4c017000 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 msgr2=0x7f6d5006c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000cd60 tx=0x7f6d40001b10 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 shutdown_connections 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6d3c060550 0x7f6d3c062a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d5013a050 0x7f6d5014d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6d500aae90 0x7f6d5014d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 --2- 172.21.15.151:0/4264355984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d50070740 0x7f6d5006c770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.669+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 >> 172.21.15.151:0/4264355984 conn(0x7f6d500b03f0 msgr2=0x7f6d500b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.673+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 shutdown_connections 2023-09-30T01:59:21.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:21.673+0000 7f6d56b83700 1 -- 172.21.15.151:0/4264355984 wait complete. 2023-09-30T01:59:24.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:24.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 Processor -- start 2023-09-30T01:59:24.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 -- start start 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 0x7f823006e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82300ae6e0 con 0x7f82300b0770 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823013d580 con 0x7f82300b0b40 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823006bbd0 con 0x7f82300afc50 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8234cb2700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f82300b0770 0x7f823013da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58452/0 (socket says 172.21.15.151:58452) 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8234cb2700 1 -- 172.21.15.151:0/927597107 learned_addr learned my addr 172.21.15.151:0/927597107 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f82354b3700 1 --2- 172.21.15.151:0/927597107 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 0x7f823006e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f822f7fe700 1 -- 172.21.15.151:0/927597107 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547689112 0 0) 0x7f82300ae6e0 con 0x7f82300b0770 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f822f7fe700 1 -- 172.21.15.151:0/927597107 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c003570 con 0x7f82300b0770 2023-09-30T01:59:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f82354b3700 1 -- 172.21.15.151:0/927597107 >> v1:172.21.15.172:6789/0 conn(0x7f82300b0b40 legacy=0x7f82300aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:24.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f82354b3700 1 -- 172.21.15.151:0/927597107 >> v1:172.21.15.151:6789/0 conn(0x7f82300b0770 legacy=0x7f823013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:24.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f82354b3700 1 -- 172.21.15.151:0/927597107 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823006cb70 con 0x7f82300afc50 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f82354b3700 1 --2- 172.21.15.151:0/927597107 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 0x7f823006e030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8228009910 tx=0x7f82280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21008c3751e9520c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f822f7fe700 1 -- 172.21.15.151:0/927597107 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822800b680 con 0x7f82300afc50 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f822f7fe700 1 -- 172.21.15.151:0/927597107 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822800b7e0 con 0x7f82300afc50 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f822f7fe700 1 -- 172.21.15.151:0/927597107 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822800bb80 con 0x7f82300afc50 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.793+0000 7f8236f16700 1 -- 172.21.15.151:0/927597107 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 msgr2=0x7f823006e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 --2- 172.21.15.151:0/927597107 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 0x7f823006e030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8228009910 tx=0x7f82280091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:24.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- 172.21.15.151:0/927597107 shutdown_connections 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 --2- 172.21.15.151:0/927597107 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300afc50 0x7f823006e030 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- 172.21.15.151:0/927597107 >> 172.21.15.151:0/927597107 conn(0x7f8230070510 msgr2=0x7f82300708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- 172.21.15.151:0/927597107 shutdown_connections 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- 172.21.15.151:0/927597107 wait complete. 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 Processor -- start 2023-09-30T01:59:24.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- start start 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82300afc50 0x7f82301504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 0x7f82301509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300b3150 con 0x7f82300b0b40 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300abab0 con 0x7f82300afc50 2023-09-30T01:59:24.803 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8236f16700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8230150f50 con 0x7f82300b0770 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49976/0 (socket says 172.21.15.151:49976) 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 -- 172.21.15.151:0/2317234764 learned_addr learned my addr 172.21.15.151:0/2317234764 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f8234cb2700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82300afc50 0x7f82301504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f822ffff700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 0x7f82301509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:24.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 -- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 msgr2=0x7f82301509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 0x7f82301509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 -- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82300afc50 msgr2=0x7f82301504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82300afc50 0x7f82301504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.797+0000 7f82354b3700 1 -- 172.21.15.151:0/2317234764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009420 con 0x7f82300b0b40 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f822ffff700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 0x7f82301509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f82354b3700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8228013000 tx=0x7f8228017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:24.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822801a430 con 0x7f82300b0b40 2023-09-30T01:59:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822801a590 con 0x7f82300b0b40 2023-09-30T01:59:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82300aab00 con 0x7f82300b0b40 2023-09-30T01:59:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822801a950 con 0x7f82300b0b40 2023-09-30T01:59:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82300b3d00 con 0x7f82300b0b40 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82300b32c0 con 0x7f82300b0b40 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.801+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f822801ad00 con 0x7f82300b0b40 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.805+0000 7f822cff9700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 0x7f8210062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.805+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f822807e290 con 0x7f82300b0b40 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.805+0000 7f8234cb2700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 0x7f8210062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.805+0000 7f8234cb2700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 0x7f8210062a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8220005c50 tx=0x7f8220009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:24.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:24.809+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f82300b32c0 con 0x7f82300b0b40 2023-09-30T01:59:25.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.005+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82300b32c0 con 0x7f82300b0b40 2023-09-30T01:59:25.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.005+0000 7f822cff9700 1 -- 172.21.15.151:0/2317234764 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f82300b32c0 con 0x7f82300b0b40 2023-09-30T01:59:25.009 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:25.009 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 msgr2=0x7f8210062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 0x7f8210062a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8220005c50 tx=0x7f8220009040 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 msgr2=0x7f82300b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8228013000 tx=0x7f8228017000 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 shutdown_connections 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8210060550 0x7f8210062a00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82300b0b40 0x7f82300b35a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82300b0770 0x7f82301509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 --2- 172.21.15.151:0/2317234764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82300afc50 0x7f82301504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 >> 172.21.15.151:0/2317234764 conn(0x7f8230070510 msgr2=0x7f823006b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 shutdown_connections 2023-09-30T01:59:25.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:25.009+0000 7f8236f16700 1 -- 172.21.15.151:0/2317234764 wait complete. 2023-09-30T01:59:28.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:28.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 Processor -- start 2023-09-30T01:59:28.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 -- start start 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf940b0070 0x7fcf9406ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf9413a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940abe30 con 0x7fcf940b0440 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9406d690 con 0x7fcf940b0070 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf9b33e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9406d970 con 0x7fcf9406e070 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf988d9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf9413a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf990da700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf940b0070 0x7fcf9406ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf988d9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf9413a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:57286/0 (socket says 172.21.15.151:57286) 2023-09-30T01:59:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf998db700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fcf940b0440 0x7fcf9406bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58464/0 (socket says 172.21.15.151:58464) 2023-09-30T01:59:28.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf990da700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf940b0070 0x7fcf9406ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49982/0 (socket says 172.21.15.151:49982) 2023-09-30T01:59:28.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.133+0000 7fcf988d9700 1 -- 172.21.15.151:0/3885824477 learned_addr learned my addr 172.21.15.151:0/3885824477 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf990da700 1 -- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 msgr2=0x7fcf9413a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf990da700 1 --2- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf9413a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf990da700 1 -- 172.21.15.151:0/3885824477 >> v1:172.21.15.151:6789/0 conn(0x7fcf940b0440 legacy=0x7fcf9406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf8bfff700 1 -- 172.21.15.151:0/3885824477 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499409931 0 0) 0x7fcf940abe30 con 0x7fcf940b0440 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf990da700 1 -- 172.21.15.151:0/3885824477 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9406dc50 con 0x7fcf940b0070 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf990da700 1 --2- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9406ea00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098e0 tx=0x7fcf900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44baaf46e5a90af9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf8bfff700 1 -- 172.21.15.151:0/3885824477 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000b640 con 0x7fcf940b0070 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf8bfff700 1 -- 172.21.15.151:0/3885824477 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9000b7a0 con 0x7fcf940b0070 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf8bfff700 1 -- 172.21.15.151:0/3885824477 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000bb40 con 0x7fcf940b0070 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 msgr2=0x7fcf9406ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9406ea00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098e0 tx=0x7fcf900091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- 172.21.15.151:0/3885824477 shutdown_connections 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf9413a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/3885824477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9406ea00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- 172.21.15.151:0/3885824477 >> 172.21.15.151:0/3885824477 conn(0x7fcf94070fe0 msgr2=0x7fcf940af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- 172.21.15.151:0/3885824477 shutdown_connections 2023-09-30T01:59:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- 172.21.15.151:0/3885824477 wait complete. 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 Processor -- start 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- start start 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf940a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf940b0440 0x7fcf94136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940a6ad0 con 0x7fcf940b0070 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940a57e0 con 0x7fcf940b0440 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf9b33e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940a5950 con 0x7fcf9406e070 2023-09-30T01:59:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.137+0000 7fcf988d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49986/0 (socket says 172.21.15.151:49986) 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf990da700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf940a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 -- 172.21.15.151:0/1915746348 learned_addr learned my addr 172.21.15.151:0/1915746348 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf998db700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf940b0440 0x7fcf94136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 -- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 msgr2=0x7fcf940a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf940a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 -- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf940b0440 msgr2=0x7fcf94136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf940b0440 0x7fcf94136ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 -- 172.21.15.151:0/1915746348 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009420 con 0x7fcf940b0070 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf990da700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf940a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf988d9700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400ea50 tx=0x7fcf84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400c640 con 0x7fcf940b0070 2023-09-30T01:59:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8400c7a0 con 0x7fcf940b0070 2023-09-30T01:59:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf94137020 con 0x7fcf940b0070 2023-09-30T01:59:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400cb20 con 0x7fcf940b0070 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf94137570 con 0x7fcf940b0070 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.141+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf940a68b0 con 0x7fcf940b0070 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.145+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fcf84013470 con 0x7fcf940b0070 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.145+0000 7fcf897fa700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 0x7fcf78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.145+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcf8407c6f0 con 0x7fcf940b0070 2023-09-30T01:59:28.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.145+0000 7fcf990da700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 0x7fcf78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.145+0000 7fcf990da700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 0x7fcf78062a00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf900091e0 tx=0x7fcf90017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.156 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.153+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcf8404ced0 con 0x7fcf940b0070 2023-09-30T01:59:28.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.349+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf940a5b10 con 0x7fcf940b0070 2023-09-30T01:59:28.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.349+0000 7fcf897fa700 1 -- 172.21.15.151:0/1915746348 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcf840504f0 con 0x7fcf940b0070 2023-09-30T01:59:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:28.353 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 msgr2=0x7fcf78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 0x7fcf78062a00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf900091e0 tx=0x7fcf90017000 comp rx=0 tx=0).stop 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 msgr2=0x7fcf9413d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400ea50 tx=0x7fcf84009250 comp rx=0 tx=0).stop 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 shutdown_connections 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcf78060550 0x7fcf78062a00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf940b0440 0x7fcf94136ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf940b0070 0x7fcf9413d270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 --2- 172.21.15.151:0/1915746348 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf9406e070 0x7fcf940a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 >> 172.21.15.151:0/1915746348 conn(0x7fcf94070fe0 msgr2=0x7fcf940af940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 shutdown_connections 2023-09-30T01:59:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:28.353+0000 7fcf9b33e700 1 -- 172.21.15.151:0/1915746348 wait complete. 2023-09-30T01:59:31.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 Processor -- start 2023-09-30T01:59:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- start start 2023-09-30T01:59:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3401408e0 0x7fd340140d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3400accb0 con 0x7fd3400b1ad0 2023-09-30T01:59:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3400ad000 con 0x7fd3401408e0 2023-09-30T01:59:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400add40 con 0x7fd3400ac8a0 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33638/0 (socket says 172.21.15.151:33638) 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 -- 172.21.15.151:0/2832400869 learned_addr learned my addr 172.21.15.151:0/2832400869 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34597b700 1 --2- 172.21.15.151:0/2832400869 >> v2:172.21.15.172:3300/0 conn(0x7fd3401408e0 0x7fd340140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 -- 172.21.15.151:0/2832400869 >> v2:172.21.15.172:3300/0 conn(0x7fd3401408e0 msgr2=0x7fd340140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 --2- 172.21.15.151:0/2832400869 >> v2:172.21.15.172:3300/0 conn(0x7fd3401408e0 0x7fd340140d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 -- 172.21.15.151:0/2832400869 >> v1:172.21.15.172:6789/0 conn(0x7fd3400b1ad0 legacy=0x7fd34006a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 -- 172.21.15.151:0/2832400869 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3400ae020 con 0x7fd3400ac8a0 2023-09-30T01:59:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd344979700 1 --2- 172.21.15.151:0/2832400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3300099b0 tx=0x7fd330009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3019bc81164b00 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd337fff700 1 -- 172.21.15.151:0/2832400869 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000b860 con 0x7fd3400ac8a0 2023-09-30T01:59:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd337fff700 1 -- 172.21.15.151:0/2832400869 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33000b9c0 con 0x7fd3400ac8a0 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd337fff700 1 -- 172.21.15.151:0/2832400869 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000bd60 con 0x7fd3400ac8a0 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- 172.21.15.151:0/2832400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 msgr2=0x7fd34006a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 --2- 172.21.15.151:0/2832400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3300099b0 tx=0x7fd330009250 comp rx=0 tx=0).stop 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- 172.21.15.151:0/2832400869 shutdown_connections 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 --2- 172.21.15.151:0/2832400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3400ac8a0 0x7fd34006a940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 --2- 172.21.15.151:0/2832400869 >> v2:172.21.15.172:3300/0 conn(0x7fd3401408e0 0x7fd340140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- 172.21.15.151:0/2832400869 >> 172.21.15.151:0/2832400869 conn(0x7fd340070510 msgr2=0x7fd340070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:31.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- 172.21.15.151:0/2832400869 shutdown_connections 2023-09-30T01:59:31.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.477+0000 7fd34617c700 1 -- 172.21.15.151:0/2832400869 wait complete. 2023-09-30T01:59:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 Processor -- start 2023-09-30T01:59:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- start start 2023-09-30T01:59:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3400ac8a0 0x7fd34013b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 0x7fd34014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34013a9e0 con 0x7fd3400b1ad0 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400a9570 con 0x7fd3400ac8a0 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400a9850 con 0x7fd3401408e0 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34597b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 0x7fd34014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47664/0 (socket says 172.21.15.151:47664) 2023-09-30T01:59:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 -- 172.21.15.151:0/3543742903 learned_addr learned my addr 172.21.15.151:0/3543742903 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34517a700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3400ac8a0 0x7fd34013b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 -- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 msgr2=0x7fd34014c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 0x7fd34014c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 -- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3400ac8a0 msgr2=0x7fd34013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3400ac8a0 0x7fd34013b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 -- 172.21.15.151:0/3543742903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300094f0 con 0x7fd3400b1ad0 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34597b700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 0x7fd34014c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd344979700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd3400ade80 tx=0x7fd330017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330011e40 con 0x7fd3400b1ad0 2023-09-30T01:59:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330024440 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34013ae70 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330024800 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.481+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34014d060 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34013a7b0 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd33001a420 con 0x7fd3400b1ad0 2023-09-30T01:59:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd3357fa700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 0x7fd324062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd33007e340 con 0x7fd3400b1ad0 2023-09-30T01:59:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd34517a700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 0x7fd324062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.485+0000 7fd34517a700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 0x7fd324062a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd33c003a80 tx=0x7fd33c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:31.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.493+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd33004eb20 con 0x7fd3400b1ad0 2023-09-30T01:59:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd34014d280 con 0x7fd3400b1ad0 2023-09-30T01:59:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd3357fa700 1 -- 172.21.15.151:0/3543742903 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd330052140 con 0x7fd3400b1ad0 2023-09-30T01:59:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:31.688 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 msgr2=0x7fd324062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 0x7fd324062a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd33c003a80 tx=0x7fd33c007ec0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 msgr2=0x7fd34013e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd3400ade80 tx=0x7fd330017000 comp rx=0 tx=0).stop 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 shutdown_connections 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd324060550 0x7fd324062a00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3401408e0 0x7fd34014c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3400b1ad0 0x7fd34013e6a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 --2- 172.21.15.151:0/3543742903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3400ac8a0 0x7fd34013b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 >> 172.21.15.151:0/3543742903 conn(0x7fd340070510 msgr2=0x7fd3400b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 shutdown_connections 2023-09-30T01:59:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:31.685+0000 7fd34617c700 1 -- 172.21.15.151:0/3543742903 wait complete. 2023-09-30T01:59:34.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 Processor -- start 2023-09-30T01:59:34.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- start start 2023-09-30T01:59:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f08000ac140 0x7f08000aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f08000ac510 0x7f080006e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000abbd0 con 0x7f08000ac140 2023-09-30T01:59:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0800138c00 con 0x7f08000ac510 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000abdd0 con 0x7f080013afd0 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f08000ac140 0x7f08000aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07ff7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f08000ac140 0x7f08000aa110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47676/0 (socket says 172.21.15.151:47676) 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07ff7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33652/0 (socket says 172.21.15.151:33652) 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 -- 172.21.15.151:0/4236882011 learned_addr learned my addr 172.21.15.151:0/4236882011 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fe7fc700 1 --2- 172.21.15.151:0/4236882011 >> v2:172.21.15.172:3300/0 conn(0x7f08000ac510 0x7f080006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 -- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 msgr2=0x7f08000a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 -- 172.21.15.151:0/4236882011 >> v2:172.21.15.172:3300/0 conn(0x7f08000ac510 msgr2=0x7f080006e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 --2- 172.21.15.151:0/4236882011 >> v2:172.21.15.172:3300/0 conn(0x7f08000ac510 0x7f080006e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 -- 172.21.15.151:0/4236882011 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000a9b50 con 0x7f08000ac140 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07ff7fe700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07feffd700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08000ac140 0x7f08000aa110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f080006a370 tx=0x7f07f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7aa3ef1156e3dd6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fdffb700 1 -- 172.21.15.151:0/4236882011 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f000b640 con 0x7f08000ac140 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fdffb700 1 -- 172.21.15.151:0/4236882011 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f000b7a0 con 0x7f08000ac140 2023-09-30T01:59:34.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fdffb700 1 -- 172.21.15.151:0/4236882011 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f000bb60 con 0x7f08000ac140 2023-09-30T01:59:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08000ac140 msgr2=0x7f08000aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08000ac140 0x7f08000aa110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f080006a370 tx=0x7f07f0009250 comp rx=0 tx=0).stop 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- 172.21.15.151:0/4236882011 shutdown_connections 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f080013afd0 0x7f08000a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- 172.21.15.151:0/4236882011 >> v2:172.21.15.172:3300/0 conn(0x7f08000ac510 0x7f080006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 --2- 172.21.15.151:0/4236882011 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08000ac140 0x7f08000aa110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- 172.21.15.151:0/4236882011 >> 172.21.15.151:0/4236882011 conn(0x7f0800070510 msgr2=0x7f0800070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- 172.21.15.151:0/4236882011 shutdown_connections 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.813+0000 7f07fffff700 1 -- 172.21.15.151:0/4236882011 wait complete. 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 Processor -- start 2023-09-30T01:59:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 -- start start 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 0x7f080013e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 0x7f080006b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08001184e0 con 0x7f080013afd0 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080013f370 con 0x7f08000ac140 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0800117a60 con 0x7f08000ac510 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fe7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 0x7f080006b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fe7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33654/0 (socket says 172.21.15.151:33654) 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fe7fc700 1 -- 172.21.15.151:0/3677215214 learned_addr learned my addr 172.21.15.151:0/3677215214 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07feffd700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 0x7f080013e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 -- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 msgr2=0x7f080006af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 -- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 msgr2=0x7f080013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 0x7f080013e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 -- 172.21.15.151:0/3677215214 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8009a20 con 0x7f080013afd0 2023-09-30T01:59:34.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07feffd700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 0x7f080013e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:34.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fe7fc700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07ff7fe700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 0x7f080006b490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009e70 tx=0x7f07f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f800cb50 con 0x7f080013afd0 2023-09-30T01:59:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f800f040 con 0x7f080013afd0 2023-09-30T01:59:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f00094f0 con 0x7f080013afd0 2023-09-30T01:59:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.817+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8014480 con 0x7f080013afd0 2023-09-30T01:59:34.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080006cda0 con 0x7f080013afd0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0800117c80 con 0x7f080013afd0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f07f8003020 con 0x7f080013afd0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07e6ffd700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 0x7f07ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f07f807ce50 con 0x7f080013afd0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.821+0000 7f07feffd700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 0x7f07ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.825+0000 7f07feffd700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 0x7f07ec062a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f080006a370 tx=0x7f07f0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:34.829+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f07f804d590 con 0x7f080013afd0 2023-09-30T01:59:35.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08000008d0 con 0x7f080013afd0 2023-09-30T01:59:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07e6ffd700 1 -- 172.21.15.151:0/3677215214 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f07f8050bb0 con 0x7f080013afd0 2023-09-30T01:59:35.028 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:35.028 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:35.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 msgr2=0x7f07ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 0x7f07ec062a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f080006a370 tx=0x7f07f0017000 comp rx=0 tx=0).stop 2023-09-30T01:59:35.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 msgr2=0x7f080006b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 0x7f080006b490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009e70 tx=0x7f07f800b040 comp rx=0 tx=0).stop 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 shutdown_connections 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f07ec060550 0x7f07ec062a00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f080013afd0 0x7f080006b490 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08000ac510 0x7f080006af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 --2- 172.21.15.151:0/3677215214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08000ac140 0x7f080013e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.025+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 >> 172.21.15.151:0/3677215214 conn(0x7f0800070510 msgr2=0x7f080006d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.029+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 shutdown_connections 2023-09-30T01:59:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:35.029+0000 7f07fffff700 1 -- 172.21.15.151:0/3677215214 wait complete. 2023-09-30T01:59:38.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:38.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 Processor -- start 2023-09-30T01:59:38.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- start start 2023-09-30T01:59:38.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f450c0ac810 0x7f450c13ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:38.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0addd0 con 0x7f450c137d60 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c13bf50 con 0x7f450c0ae600 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0acc60 con 0x7f450c0ac810 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f450c0ac810 0x7f450c13ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f450c0ac810 0x7f450c13ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47696/0 (socket says 172.21.15.151:47696) 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 -- 172.21.15.151:0/4120826941 learned_addr learned my addr 172.21.15.151:0/4120826941 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450b59e700 1 --1- 172.21.15.151:0/4120826941 >> v1:172.21.15.151:6789/0 conn(0x7f450c137d60 0x7f450c1381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34466/0 (socket says 172.21.15.151:34466) 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 -- 172.21.15.151:0/4120826941 >> v1:172.21.15.172:6789/0 conn(0x7f450c0ae600 legacy=0x7f450c13b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 -- 172.21.15.151:0/4120826941 >> v1:172.21.15.151:6789/0 conn(0x7f450c137d60 legacy=0x7f450c1381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 -- 172.21.15.151:0/4120826941 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c0acf40 con 0x7f450c0ac810 2023-09-30T01:59:38.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f450a59c700 1 --2- 172.21.15.151:0/4120826941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c13ba10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0099b0 tx=0x7f44fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2644153ce68cc389 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:38.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f4509d9b700 1 -- 172.21.15.151:0/4120826941 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00b860 con 0x7f450c0ac810 2023-09-30T01:59:38.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f4509d9b700 1 -- 172.21.15.151:0/4120826941 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc00b9c0 con 0x7f450c0ac810 2023-09-30T01:59:38.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f4509d9b700 1 -- 172.21.15.151:0/4120826941 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00bd60 con 0x7f450c0ac810 2023-09-30T01:59:38.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- 172.21.15.151:0/4120826941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 msgr2=0x7f450c13ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 --2- 172.21.15.151:0/4120826941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c13ba10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0099b0 tx=0x7f44fc009250 comp rx=0 tx=0).stop 2023-09-30T01:59:38.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- 172.21.15.151:0/4120826941 shutdown_connections 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 --2- 172.21.15.151:0/4120826941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c13ba10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- 172.21.15.151:0/4120826941 >> 172.21.15.151:0/4120826941 conn(0x7f450c070510 msgr2=0x7f450c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- 172.21.15.151:0/4120826941 shutdown_connections 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.157+0000 7f45108f9700 1 -- 172.21.15.151:0/4120826941 wait complete. 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 Processor -- start 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- start start 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f450c0ae600 0x7f450c14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 0x7f450c13c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0a5830 con 0x7f450c0ac810 2023-09-30T01:59:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c13c1f0 con 0x7f450c137d60 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c14ea10 con 0x7f450c0ae600 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450a59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f450c0ae600 0x7f450c14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47712/0 (socket says 172.21.15.151:47712) 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 -- 172.21.15.151:0/375897523 learned_addr learned my addr 172.21.15.151:0/375897523 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:38.165 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450b59e700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 0x7f450c13c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:38.166 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 -- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f450c0ae600 msgr2=0x7f450c14e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f450c0ae600 0x7f450c14e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 -- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 msgr2=0x7f450c13c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 0x7f450c13c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 -- 172.21.15.151:0/375897523 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0094f0 con 0x7f450c0ac810 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450b59e700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 0x7f450c13c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f450ad9d700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c5c0 tx=0x7f44f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f4011590 con 0x7f450c0ac810 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f4011710 con 0x7f450c0ac810 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c13cbc0 con 0x7f450c0ac810 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f4011ad0 con 0x7f450c0ac810 2023-09-30T01:59:38.167 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.161+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c13d0b0 con 0x7f450c0ac810 2023-09-30T01:59:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c0008d0 con 0x7f450c0ac810 2023-09-30T01:59:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f44f401b3b0 con 0x7f450c0ac810 2023-09-30T01:59:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f4502ffd700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 0x7f44ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f44f407ba80 con 0x7f450c0ac810 2023-09-30T01:59:38.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f450a59c700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 0x7f44ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.165+0000 7f450a59c700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 0x7f44ec062a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0099b0 tx=0x7f44fc017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:38.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.169+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44f404c260 con 0x7f450c0ac810 2023-09-30T01:59:38.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.365+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f450c0b0c10 con 0x7f450c0ac810 2023-09-30T01:59:38.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.365+0000 7f4502ffd700 1 -- 172.21.15.151:0/375897523 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f44f4017080 con 0x7f450c0ac810 2023-09-30T01:59:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:38.370 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 msgr2=0x7f44ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 0x7f44ec062a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0099b0 tx=0x7f44fc017000 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 msgr2=0x7f450c14df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c5c0 tx=0x7f44f400c8d0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 shutdown_connections 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f44ec060550 0x7f44ec062a00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f450c137d60 0x7f450c13c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f450c0ae600 0x7f450c14e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 --2- 172.21.15.151:0/375897523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f450c0ac810 0x7f450c14df90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 >> 172.21.15.151:0/375897523 conn(0x7f450c070510 msgr2=0x7f450c0b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 shutdown_connections 2023-09-30T01:59:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:38.369+0000 7f45108f9700 1 -- 172.21.15.151:0/375897523 wait complete. 2023-09-30T01:59:41.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:41.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 Processor -- start 2023-09-30T01:59:41.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 -- start start 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11e00708e0 0x7f11e006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e00ac560 con 0x7f11e00aa470 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e006f500 con 0x7f11e00708e0 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11e4c38700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e00ab1a0 con 0x7f11e0070510 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11df7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11df7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44574/0 (socket says 172.21.15.151:44574) 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11df7fe700 1 -- 172.21.15.151:0/3352679297 learned_addr learned my addr 172.21.15.151:0/3352679297 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.493+0000 7f11deffd700 1 --2- 172.21.15.151:0/3352679297 >> v2:172.21.15.172:3300/0 conn(0x7f11e00708e0 0x7f11e006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11deffd700 1 -- 172.21.15.151:0/3352679297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 msgr2=0x7f11e006e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11deffd700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11deffd700 1 -- 172.21.15.151:0/3352679297 >> v1:172.21.15.172:6789/0 conn(0x7f11e00aa470 legacy=0x7f11e013d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11deffd700 1 -- 172.21.15.151:0/3352679297 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e00aae60 con 0x7f11e00708e0 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11de7fc700 1 -- 172.21.15.151:0/3352679297 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878269996 0 0) 0x7f11e00ac560 con 0x7f11e00aa470 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11df7fe700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:41.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11deffd700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00708e0 0x7f11e006ef00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009e60 tx=0x7f11d0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dae7fcf9fd867f4c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:41.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11de7fc700 1 -- 172.21.15.151:0/3352679297 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d000e070 con 0x7f11e00708e0 2023-09-30T01:59:41.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11de7fc700 1 -- 172.21.15.151:0/3352679297 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d000bb60 con 0x7f11e00708e0 2023-09-30T01:59:41.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11de7fc700 1 -- 172.21.15.151:0/3352679297 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0012430 con 0x7f11e00708e0 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- 172.21.15.151:0/3352679297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00708e0 msgr2=0x7f11e006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00708e0 0x7f11e006ef00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009e60 tx=0x7f11d0009780 comp rx=0 tx=0).stop 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- 172.21.15.151:0/3352679297 shutdown_connections 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00708e0 0x7f11e006ef00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- 172.21.15.151:0/3352679297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e0070510 0x7f11e006e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- 172.21.15.151:0/3352679297 >> 172.21.15.151:0/3352679297 conn(0x7f11e00b03e0 msgr2=0x7f11e00b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- 172.21.15.151:0/3352679297 shutdown_connections 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- 172.21.15.151:0/3352679297 wait complete. 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 Processor -- start 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- start start 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 0x7f11e01394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00aa470 0x7f11e0135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0136780 con 0x7f11e0070510 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0136c10 con 0x7f11e00aa470 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11e4c38700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e014cf70 con 0x7f11e014c590 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11dffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11df7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 0x7f11e01394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11dffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44578/0 (socket says 172.21.15.151:44578) 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.497+0000 7f11dffff700 1 -- 172.21.15.151:0/2489978370 learned_addr learned my addr 172.21.15.151:0/2489978370 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11deffd700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00aa470 0x7f11e0135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 -- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00aa470 msgr2=0x7f11e0135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00aa470 0x7f11e0135ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 -- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 msgr2=0x7f11e01394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 0x7f11e01394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 -- 172.21.15.151:0/2489978370 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40094f0 con 0x7f11e014c590 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11df7fe700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 0x7f11e01394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11dffff700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c5d0 tx=0x7f11d8007d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011590 con 0x7f11e014c590 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d80116f0 con 0x7f11e014c590 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d0009a20 con 0x7f11e014c590 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011ab0 con 0x7f11e014c590 2023-09-30T01:59:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e0135760 con 0x7f11e014c590 2023-09-30T01:59:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e0136530 con 0x7f11e014c590 2023-09-30T01:59:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.501+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f11d801b3b0 con 0x7f11e014c590 2023-09-30T01:59:41.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.505+0000 7f11c77fe700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 0x7f11cc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.505+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f11d807bad0 con 0x7f11e014c590 2023-09-30T01:59:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.505+0000 7f11df7fe700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 0x7f11cc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:41.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.505+0000 7f11df7fe700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 0x7f11cc062a00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006d20 tx=0x7f11d4009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:41.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.513+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f11d804c2b0 con 0x7f11e014c590 2023-09-30T01:59:41.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f11e01368f0 con 0x7f11e014c590 2023-09-30T01:59:41.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11c77fe700 1 -- 172.21.15.151:0/2489978370 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f11d8017080 con 0x7f11e014c590 2023-09-30T01:59:41.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:41.708 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 msgr2=0x7f11cc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 0x7f11cc062a00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006d20 tx=0x7f11d4009250 comp rx=0 tx=0).stop 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 msgr2=0x7f11e014ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c5d0 tx=0x7f11d8007d70 comp rx=0 tx=0).stop 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 shutdown_connections 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f11cc060550 0x7f11cc062a00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11e014c590 0x7f11e014ca30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e00aa470 0x7f11e0135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 --2- 172.21.15.151:0/2489978370 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11e0070510 0x7f11e01394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:41.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 >> 172.21.15.151:0/2489978370 conn(0x7f11e00b03e0 msgr2=0x7f11e00ad1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:41.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 shutdown_connections 2023-09-30T01:59:41.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:41.705+0000 7f11e4c38700 1 -- 172.21.15.151:0/2489978370 wait complete. 2023-09-30T01:59:44.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:44.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 Processor -- start 2023-09-30T01:59:44.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 -- start start 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5954070510 0x7f59540a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59540aae80 0x7f595413b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59540ac590 con 0x7f59540708e0 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59540ab550 con 0x7f5954070510 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5953fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59540ac870 con 0x7f59540aae80 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f59527fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f59540708e0 0x7f595413b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:36504/0 (socket says 172.21.15.151:36504) 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f59527fc700 1 -- 172.21.15.151:0/1696083984 learned_addr learned my addr 172.21.15.151:0/1696083984 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1696083984 >> v2:172.21.15.151:3300/0 conn(0x7f5954070510 0x7f59540a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f59537fe700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59540aae80 0x7f595413b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 -- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59540aae80 msgr2=0x7f595413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:44.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59540aae80 0x7f595413b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 -- 172.21.15.151:0/1696083984 >> v1:172.21.15.151:6789/0 conn(0x7f59540708e0 legacy=0x7f595413b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:44.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5951ffb700 1 -- 172.21.15.151:0/1696083984 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491440021 0 0) 0x7f59540ac590 con 0x7f59540708e0 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 -- 172.21.15.151:0/1696083984 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595413bd10 con 0x7f5954070510 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5954070510 0x7f59540a9db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5948009910 tx=0x7f59480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3629c9538a1aa77e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5951ffb700 1 -- 172.21.15.151:0/1696083984 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800b640 con 0x7f5954070510 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5951ffb700 1 -- 172.21.15.151:0/1696083984 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594800b7a0 con 0x7f5954070510 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.833+0000 7f5951ffb700 1 -- 172.21.15.151:0/1696083984 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800bb60 con 0x7f5954070510 2023-09-30T01:59:44.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5954070510 msgr2=0x7f59540a9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:44.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5954070510 0x7f59540a9db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5948009910 tx=0x7f59480091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- 172.21.15.151:0/1696083984 shutdown_connections 2023-09-30T01:59:44.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59540aae80 0x7f595413b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 --2- 172.21.15.151:0/1696083984 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5954070510 0x7f59540a9db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- 172.21.15.151:0/1696083984 >> 172.21.15.151:0/1696083984 conn(0x7f59540b03e0 msgr2=0x7f59540b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:44.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- 172.21.15.151:0/1696083984 shutdown_connections 2023-09-30T01:59:44.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- 172.21.15.151:0/1696083984 wait complete. 2023-09-30T01:59:44.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 Processor -- start 2023-09-30T01:59:44.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.837+0000 7f5953fff700 1 -- start start 2023-09-30T01:59:44.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 0x7f595413c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59540aae80 0x7f595413e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595413d040 con 0x7f59540708e0 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59540a54c0 con 0x7f59540aae80 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595413cb50 con 0x7f5954070510 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 0x7f595413c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44588/0 (socket says 172.21.15.151:44588) 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5952ffd700 1 -- 172.21.15.151:0/1335049938 learned_addr learned my addr 172.21.15.151:0/1335049938 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59537fe700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59540aae80 0x7f595413e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 -- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 msgr2=0x7f59541379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 -- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59540aae80 msgr2=0x7f595413e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59540aae80 0x7f595413e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 -- 172.21.15.151:0/1335049938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5948009420 con 0x7f59540708e0 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f59527fc700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 0x7f595413c610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59440011e0 tx=0x7f5944007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944005b30 con 0x7f59540708e0 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5944005c90 con 0x7f59540708e0 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595413cdd0 con 0x7f59540708e0 2023-09-30T01:59:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944007300 con 0x7f59540708e0 2023-09-30T01:59:44.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595413e8a0 con 0x7f59540708e0 2023-09-30T01:59:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.841+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59540008d0 con 0x7f59540708e0 2023-09-30T01:59:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.845+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f594401b3b0 con 0x7f59540708e0 2023-09-30T01:59:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.845+0000 7f593affd700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 0x7f5940062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.845+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f594407b990 con 0x7f59540708e0 2023-09-30T01:59:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.845+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 0x7f5940062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:44.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.845+0000 7f5952ffd700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 0x7f5940062a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f59480093a0 tx=0x7f5948017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:44.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:44.853+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f594404c170 con 0x7f59540708e0 2023-09-30T01:59:45.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.053+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f59540ac2b0 con 0x7f59540708e0 2023-09-30T01:59:45.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.053+0000 7f593affd700 1 -- 172.21.15.151:0/1335049938 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f594404f790 con 0x7f59540708e0 2023-09-30T01:59:45.057 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:45.057 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:45.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 msgr2=0x7f5940062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 0x7f5940062a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f59480093a0 tx=0x7f5948017000 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 msgr2=0x7f595413c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 0x7f595413c610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59440011e0 tx=0x7f5944007d30 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 shutdown_connections 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5940060550 0x7f5940062a00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59540aae80 0x7f595413e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59540708e0 0x7f595413c610 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 --2- 172.21.15.151:0/1335049938 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5954070510 0x7f59541379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 >> 172.21.15.151:0/1335049938 conn(0x7f59540b03e0 msgr2=0x7f595406e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 shutdown_connections 2023-09-30T01:59:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:45.057+0000 7f5953fff700 1 -- 172.21.15.151:0/1335049938 wait complete. 2023-09-30T01:59:48.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:48.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 Processor -- start 2023-09-30T01:59:48.191 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 -- start start 2023-09-30T01:59:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbcf006e660 0x7fbcf013d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf0067e80 con 0x7fbcf0067ab0 2023-09-30T01:59:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf006da30 con 0x7fbcf006e660 2023-09-30T01:59:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcf4f34700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf006eb80 con 0x7fbcf01364e0 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44590/0 (socket says 172.21.15.151:44590) 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 -- 172.21.15.151:0/3283443638 learned_addr learned my addr 172.21.15.151:0/3283443638 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceeffd700 1 --2- 172.21.15.151:0/3283443638 >> v2:172.21.15.172:3300/0 conn(0x7fbcf006e660 0x7fbcf013d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 -- 172.21.15.151:0/3283443638 >> v2:172.21.15.172:3300/0 conn(0x7fbcf006e660 msgr2=0x7fbcf013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 --2- 172.21.15.151:0/3283443638 >> v2:172.21.15.172:3300/0 conn(0x7fbcf006e660 0x7fbcf013d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 -- 172.21.15.151:0/3283443638 >> v1:172.21.15.172:6789/0 conn(0x7fbcf0067ab0 legacy=0x7fbcf013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 -- 172.21.15.151:0/3283443638 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf013d9e0 con 0x7fbcf01364e0 2023-09-30T01:59:48.195 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbceffff700 1 --2- 172.21.15.151:0/3283443638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c0d0 tx=0x7fbce800a950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddeedc8761e889ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:48.195 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.189+0000 7fbcee7fc700 1 -- 172.21.15.151:0/3283443638 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce800f070 con 0x7fbcf01364e0 2023-09-30T01:59:48.195 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcee7fc700 1 -- 172.21.15.151:0/3283443638 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce800a2e0 con 0x7fbcf01364e0 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcee7fc700 1 -- 172.21.15.151:0/3283443638 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce800cd80 con 0x7fbcf01364e0 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- 172.21.15.151:0/3283443638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 msgr2=0x7fbcf0136960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/3283443638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c0d0 tx=0x7fbce800a950 comp rx=0 tx=0).stop 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- 172.21.15.151:0/3283443638 shutdown_connections 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/3283443638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf01364e0 0x7fbcf0136960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/3283443638 >> v2:172.21.15.172:3300/0 conn(0x7fbcf006e660 0x7fbcf013d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- 172.21.15.151:0/3283443638 >> 172.21.15.151:0/3283443638 conn(0x7fbcf00b03f0 msgr2=0x7fbcf00b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- 172.21.15.151:0/3283443638 shutdown_connections 2023-09-30T01:59:48.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- 172.21.15.151:0/3283443638 wait complete. 2023-09-30T01:59:48.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 Processor -- start 2023-09-30T01:59:48.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- start start 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 0x7fbcf00a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf01364e0 0x7fbcf014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf014efa0 con 0x7fbcf0067ab0 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00a5940 con 0x7fbcf01364e0 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbcf4f34700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf013a100 con 0x7fbcf006e660 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbceeffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.193+0000 7fbceeffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44594/0 (socket says 172.21.15.151:44594) 2023-09-30T01:59:48.199 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 -- 172.21.15.151:0/2099811148 learned_addr learned my addr 172.21.15.151:0/2099811148 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcef7fe700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 0x7fbcf00a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceffff700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf01364e0 0x7fbcf014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 -- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf01364e0 msgr2=0x7fbcf014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf01364e0 0x7fbcf014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 -- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 msgr2=0x7fbcf00a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 0x7fbcf00a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 -- 172.21.15.151:0/2099811148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce800abf0 con 0x7fbcf006e660 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcef7fe700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 0x7fbcf00a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:48.200 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbceeffd700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbce400ecb0 tx=0x7fbce40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:48.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce400c850 con 0x7fbcf006e660 2023-09-30T01:59:48.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce400c9d0 con 0x7fbcf006e660 2023-09-30T01:59:48.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf00a5dd0 con 0x7fbcf006e660 2023-09-30T01:59:48.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce400cd90 con 0x7fbcf006e660 2023-09-30T01:59:48.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf00a50d0 con 0x7fbcf006e660 2023-09-30T01:59:48.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.197+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf00008d0 con 0x7fbcf006e660 2023-09-30T01:59:48.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.201+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fbce4013680 con 0x7fbcf006e660 2023-09-30T01:59:48.203 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.201+0000 7fbcd77fe700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 0x7fbcdc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:48.203 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.201+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fbce407c6b0 con 0x7fbcf006e660 2023-09-30T01:59:48.203 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.201+0000 7fbcef7fe700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 0x7fbcdc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.201+0000 7fbcef7fe700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 0x7fbcdc062a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001dc0 tx=0x7fbce0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:48.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.209+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbce404cdf0 con 0x7fbcf006e660 2023-09-30T01:59:48.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbcf00a50d0 con 0x7fbcf006e660 2023-09-30T01:59:48.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcd77fe700 1 -- 172.21.15.151:0/2099811148 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbcf00a50d0 con 0x7fbcf006e660 2023-09-30T01:59:48.410 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:48.410 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 msgr2=0x7fbcdc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 0x7fbcdc062a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001dc0 tx=0x7fbce0009040 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 msgr2=0x7fbcf0139ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbce400ecb0 tx=0x7fbce40091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 shutdown_connections 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbcdc060550 0x7fbcdc062a00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcf01364e0 0x7fbcf014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbcf006e660 0x7fbcf0139ac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 --2- 172.21.15.151:0/2099811148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbcf0067ab0 0x7fbcf00a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:48.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 >> 172.21.15.151:0/2099811148 conn(0x7fbcf00b03f0 msgr2=0x7fbcf0132770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 shutdown_connections 2023-09-30T01:59:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:48.405+0000 7fbcf4f34700 1 -- 172.21.15.151:0/2099811148 wait complete. 2023-09-30T01:59:51.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:51.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 Processor -- start 2023-09-30T01:59:51.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 -- start start 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb930139f20 0x7fb9300aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93006c3f0 con 0x7fb93006ea20 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb93006db10 con 0x7fb930139f20 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb934f5b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300b0e50 con 0x7fb9300b0a80 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92f7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92effd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb93006ea20 0x7fb9300aa4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:46096/0 (socket says 172.21.15.151:46096) 2023-09-30T01:59:51.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92effd700 1 -- 172.21.15.151:0/3439407533 learned_addr learned my addr 172.21.15.151:0/3439407533 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T01:59:51.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92ffff700 1 --2- 172.21.15.151:0/3439407533 >> v2:172.21.15.172:3300/0 conn(0x7fb930139f20 0x7fb9300aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92e7fc700 1 -- 172.21.15.151:0/3439407533 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533351349 0 0) 0x7fb93006c3f0 con 0x7fb93006ea20 2023-09-30T01:59:51.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.528+0000 7fb92e7fc700 1 -- 172.21.15.151:0/3439407533 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb918003570 con 0x7fb93006ea20 2023-09-30T01:59:51.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92f7fe700 1 -- 172.21.15.151:0/3439407533 >> v2:172.21.15.172:3300/0 conn(0x7fb930139f20 msgr2=0x7fb9300aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:51.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92f7fe700 1 --2- 172.21.15.151:0/3439407533 >> v2:172.21.15.172:3300/0 conn(0x7fb930139f20 0x7fb9300aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92f7fe700 1 -- 172.21.15.151:0/3439407533 >> v1:172.21.15.151:6789/0 conn(0x7fb93006ea20 legacy=0x7fb9300aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92f7fe700 1 -- 172.21.15.151:0/3439407533 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93006dd80 con 0x7fb9300b0a80 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92f7fe700 1 --2- 172.21.15.151:0/3439407533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93006e4e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb9240098e0 tx=0x7fb9240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930afbeaa01833a8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92e7fc700 1 -- 172.21.15.151:0/3439407533 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9240108a0 con 0x7fb9300b0a80 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92e7fc700 1 -- 172.21.15.151:0/3439407533 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924010a00 con 0x7fb9300b0a80 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb92e7fc700 1 -- 172.21.15.151:0/3439407533 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924010da0 con 0x7fb9300b0a80 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- 172.21.15.151:0/3439407533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 msgr2=0x7fb93006e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 --2- 172.21.15.151:0/3439407533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93006e4e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb9240098e0 tx=0x7fb9240091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- 172.21.15.151:0/3439407533 shutdown_connections 2023-09-30T01:59:51.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 --2- 172.21.15.151:0/3439407533 >> v2:172.21.15.172:3300/0 conn(0x7fb930139f20 0x7fb9300aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 --2- 172.21.15.151:0/3439407533 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93006e4e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- 172.21.15.151:0/3439407533 >> 172.21.15.151:0/3439407533 conn(0x7fb930070510 msgr2=0x7fb930070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:51.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- 172.21.15.151:0/3439407533 shutdown_connections 2023-09-30T01:59:51.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- 172.21.15.151:0/3439407533 wait complete. 2023-09-30T01:59:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 Processor -- start 2023-09-30T01:59:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.532+0000 7fb934f5b700 1 -- start start 2023-09-30T01:59:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb93006ea20 0x7fb93013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb930139f20 0x7fb930137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93014ca40 con 0x7fb93006ea20 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93013e1b0 con 0x7fb930139f20 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93014c4b0 con 0x7fb9300b0a80 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92f7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb93006ea20 0x7fb93013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52032/0 (socket says 172.21.15.151:52032) 2023-09-30T01:59:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 -- 172.21.15.151:0/1859019502 learned_addr learned my addr 172.21.15.151:0/1859019502 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:51.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92ffff700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb930139f20 0x7fb930137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 -- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb930139f20 msgr2=0x7fb930137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb930139f20 0x7fb930137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 -- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb93006ea20 msgr2=0x7fb93013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb93006ea20 0x7fb93013dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 -- 172.21.15.151:0/1859019502 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb924009420 con 0x7fb9300b0a80 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb92effd700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb92000c7c0 tx=0x7fb920007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920005cd0 con 0x7fb9300b0a80 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb920005e30 con 0x7fb9300b0a80 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93014c6d0 con 0x7fb9300b0a80 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920007320 con 0x7fb9300b0a80 2023-09-30T01:59:51.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb930138580 con 0x7fb9300b0a80 2023-09-30T01:59:51.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.536+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93014cbf0 con 0x7fb9300b0a80 2023-09-30T01:59:51.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.540+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb92001b3b0 con 0x7fb9300b0a80 2023-09-30T01:59:51.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.540+0000 7fb9177fe700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 0x7fb91c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:51.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.540+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb92007bc50 con 0x7fb9300b0a80 2023-09-30T01:59:51.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.540+0000 7fb92f7fe700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 0x7fb91c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:51.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.540+0000 7fb92f7fe700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 0x7fb91c062a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb92401b040 tx=0x7fb924018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:51.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.544+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb93014cbf0 con 0x7fb9300b0a80 2023-09-30T01:59:51.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.744+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb93014cbf0 con 0x7fb9300b0a80 2023-09-30T01:59:51.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.744+0000 7fb9177fe700 1 -- 172.21.15.151:0/1859019502 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb93014cbf0 con 0x7fb9300b0a80 2023-09-30T01:59:51.749 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:51.749 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 msgr2=0x7fb91c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 0x7fb91c062a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb92401b040 tx=0x7fb924018000 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 msgr2=0x7fb93014bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb92000c7c0 tx=0x7fb920007e00 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 shutdown_connections 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fb91c060550 0x7fb91c062a00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb930139f20 0x7fb930137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9300b0a80 0x7fb93014bf70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 --2- 172.21.15.151:0/1859019502 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb93006ea20 0x7fb93013dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:51.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 >> 172.21.15.151:0/1859019502 conn(0x7fb930070510 msgr2=0x7fb930136620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:51.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 shutdown_connections 2023-09-30T01:59:51.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:51.748+0000 7fb934f5b700 1 -- 172.21.15.151:0/1859019502 wait complete. 2023-09-30T01:59:54.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 Processor -- start 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 -- start start 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 0x7f776406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77640adf40 con 0x7f776406ea60 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776406b590 con 0x7f776406b770 2023-09-30T01:59:54.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f776ae69700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776406d020 con 0x7f77640b44f0 2023-09-30T01:59:54.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769666700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 0x7f776406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769666700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59476/0 (socket says 172.21.15.151:59476) 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769666700 1 -- 172.21.15.151:0/2882927138 learned_addr learned my addr 172.21.15.151:0/2882927138 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 -- 172.21.15.151:0/2882927138 >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 msgr2=0x7f7764140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 --2- 172.21.15.151:0/2882927138 >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 -- 172.21.15.151:0/2882927138 >> v1:172.21.15.151:6789/0 conn(0x7f776406ea60 legacy=0x7f776406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7768e65700 1 -- 172.21.15.151:0/2882927138 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295505486 0 0) 0x7f77640adf40 con 0x7f776406ea60 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 -- 172.21.15.151:0/2882927138 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77640b48c0 con 0x7f77640b44f0 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769666700 1 --2- 172.21.15.151:0/2882927138 >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:54.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7769e67700 1 --2- 172.21.15.151:0/2882927138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 0x7f776406b050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f77540098e0 tx=0x7f77540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8864cd430260f75 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7768e65700 1 -- 172.21.15.151:0/2882927138 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400b640 con 0x7f77640b44f0 2023-09-30T01:59:54.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7768e65700 1 -- 172.21.15.151:0/2882927138 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400b7a0 con 0x7f77640b44f0 2023-09-30T01:59:54.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.876+0000 7f7768e65700 1 -- 172.21.15.151:0/2882927138 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400bb40 con 0x7f77640b44f0 2023-09-30T01:59:54.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- 172.21.15.151:0/2882927138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 msgr2=0x7f776406b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- 172.21.15.151:0/2882927138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 0x7f776406b050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f77540098e0 tx=0x7f77540091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- 172.21.15.151:0/2882927138 shutdown_connections 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- 172.21.15.151:0/2882927138 >> v2:172.21.15.151:3300/0 conn(0x7f776406b770 0x7f7764140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- 172.21.15.151:0/2882927138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77640b44f0 0x7f776406b050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- 172.21.15.151:0/2882927138 >> 172.21.15.151:0/2882927138 conn(0x7f7764070fe0 msgr2=0x7f77640b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- 172.21.15.151:0/2882927138 shutdown_connections 2023-09-30T01:59:54.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- 172.21.15.151:0/2882927138 wait complete. 2023-09-30T01:59:54.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 Processor -- start 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- start start 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f776406ea60 0x7f77640a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77640b44f0 0x7f776414d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776414d2e0 con 0x7f77640b44f0 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776413d550 con 0x7f776406ea60 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776ae69700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776413d710 con 0x7f776406b770 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f776a668700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77640b44f0 0x7f776414d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52042/0 (socket says 172.21.15.151:52042) 2023-09-30T01:59:54.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 -- 172.21.15.151:0/3043503596 learned_addr learned my addr 172.21.15.151:0/3043503596 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:54.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769666700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f776406ea60 0x7f77640a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 -- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f776406ea60 msgr2=0x7f77640a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f776406ea60 0x7f77640a89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 -- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77640b44f0 msgr2=0x7f776414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.880+0000 7f7769e67700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77640b44f0 0x7f776414d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7769e67700 1 -- 172.21.15.151:0/3043503596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760009ae0 con 0x7f776406b770 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7769e67700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f776406d160 tx=0x7f7754017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400b910 con 0x7f776406b770 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400ba70 con 0x7f776406b770 2023-09-30T01:59:54.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7754009420 con 0x7f776406b770 2023-09-30T01:59:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754011670 con 0x7f776406b770 2023-09-30T01:59:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776413b790 con 0x7f776406b770 2023-09-30T01:59:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77640008d0 con 0x7f776406b770 2023-09-30T01:59:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7754011aa0 con 0x7f776406b770 2023-09-30T01:59:54.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 0x7f7748062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.884+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f775407dda0 con 0x7f776406b770 2023-09-30T01:59:54.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.888+0000 7f7769666700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 0x7f7748062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.888+0000 7f7769666700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 0x7f7748062a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77641414e0 tx=0x7f776000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:54.892+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f775404e580 con 0x7f776406b770 2023-09-30T01:59:55.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.088+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f776406d290 con 0x7f776406b770 2023-09-30T01:59:55.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.092+0000 7f7759ffb700 1 -- 172.21.15.151:0/3043503596 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7754051ba0 con 0x7f776406b770 2023-09-30T01:59:55.097 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:55.097 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 msgr2=0x7f7748062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 0x7f7748062a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f77641414e0 tx=0x7f776000b040 comp rx=0 tx=0).stop 2023-09-30T01:59:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 msgr2=0x7f77640a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f776406d160 tx=0x7f7754017000 comp rx=0 tx=0).stop 2023-09-30T01:59:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 shutdown_connections 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7748060550 0x7f7748062a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77640b44f0 0x7f776414d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f776406ea60 0x7f77640a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 --2- 172.21.15.151:0/3043503596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f776406b770 0x7f77640a9300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 >> 172.21.15.151:0/3043503596 conn(0x7f7764070fe0 msgr2=0x7f776406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 shutdown_connections 2023-09-30T01:59:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:55.096+0000 7f776ae69700 1 -- 172.21.15.151:0/3043503596 wait complete. 2023-09-30T01:59:58.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:58.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.220+0000 7fe966992700 1 Processor -- start 2023-09-30T01:59:58.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.220+0000 7fe966992700 1 -- start start 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 0x7fe96013d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe96013dca0 0x7fe96013e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600aa210 con 0x7fe9600b0a20 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96013e610 con 0x7fe96013dca0 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600a8fb0 con 0x7fe960141010 2023-09-30T01:59:58.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe96013dca0 0x7fe96013e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe96013dca0 0x7fe96013e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59490/0 (socket says 172.21.15.151:59490) 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 -- 172.21.15.151:0/1849947937 learned_addr learned my addr 172.21.15.151:0/1849947937 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe96518f700 1 --2- 172.21.15.151:0/1849947937 >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 0x7fe96013d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 -- 172.21.15.151:0/1849947937 >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 msgr2=0x7fe96013d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 --2- 172.21.15.151:0/1849947937 >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 0x7fe96013d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 -- 172.21.15.151:0/1849947937 >> v1:172.21.15.172:6789/0 conn(0x7fe9600b0a20 legacy=0x7fe960140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 -- 172.21.15.151:0/1849947937 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600b0df0 con 0x7fe96013dca0 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe96518f700 1 --2- 172.21.15.151:0/1849947937 >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 0x7fe96013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:58.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966191700 1 --2- 172.21.15.151:0/1849947937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe96013dca0 0x7fe96013e0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe9600b1060 tx=0x7fe95c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce8ed530cc8c122 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe96498e700 1 -- 172.21.15.151:0/1849947937 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c00b820 con 0x7fe96013dca0 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe96498e700 1 -- 172.21.15.151:0/1849947937 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c00b980 con 0x7fe96013dca0 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe96498e700 1 -- 172.21.15.151:0/1849947937 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c00bd20 con 0x7fe96013dca0 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- 172.21.15.151:0/1849947937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe96013dca0 msgr2=0x7fe96013e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 --2- 172.21.15.151:0/1849947937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe96013dca0 0x7fe96013e0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe9600b1060 tx=0x7fe95c009250 comp rx=0 tx=0).stop 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- 172.21.15.151:0/1849947937 shutdown_connections 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 --2- 172.21.15.151:0/1849947937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe96013dca0 0x7fe96013e0d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 --2- 172.21.15.151:0/1849947937 >> v2:172.21.15.172:3300/0 conn(0x7fe960141010 0x7fe96013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- 172.21.15.151:0/1849947937 >> 172.21.15.151:0/1849947937 conn(0x7fe960070510 msgr2=0x7fe960070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- 172.21.15.151:0/1849947937 shutdown_connections 2023-09-30T01:59:58.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- 172.21.15.151:0/1849947937 wait complete. 2023-09-30T01:59:58.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 Processor -- start 2023-09-30T01:59:58.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.224+0000 7fe966992700 1 -- start start 2023-09-30T01:59:58.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 0x7fe960118ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe96013dca0 0x7fe9601429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960119470 con 0x7fe9600b0a20 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96011ab50 con 0x7fe96013dca0 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96011ae30 con 0x7fe960141010 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52046/0 (socket says 172.21.15.151:52046) 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 -- 172.21.15.151:0/3536566703 learned_addr learned my addr 172.21.15.151:0/3536566703 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe965990700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 0x7fe960118ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe96518f700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe96013dca0 0x7fe9601429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 -- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe96013dca0 msgr2=0x7fe9601429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe96013dca0 0x7fe9601429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 -- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 msgr2=0x7fe960118ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 0x7fe960118ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 -- 172.21.15.151:0/3536566703 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0094f0 con 0x7fe960141010 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe965990700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 0x7fe960118ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966191700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009290 tx=0x7fe95c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c01b520 con 0x7fe960141010 2023-09-30T01:59:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c020070 con 0x7fe960141010 2023-09-30T01:59:58.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96011b110 con 0x7fe960141010 2023-09-30T01:59:58.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c01b8f0 con 0x7fe960141010 2023-09-30T01:59:58.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960144360 con 0x7fe960141010 2023-09-30T01:59:58.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.228+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96011b2f0 con 0x7fe960141010 2023-09-30T01:59:58.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.232+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe95c022030 con 0x7fe960141010 2023-09-30T01:59:58.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.232+0000 7fe955ffb700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 0x7fe944062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.232+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe95c07e800 con 0x7fe960141010 2023-09-30T01:59:58.237 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.232+0000 7fe965990700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 0x7fe944062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.237 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.232+0000 7fe965990700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 0x7fe944062a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe958007fd0 tx=0x7fe958009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.242 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.240+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe95c04ef40 con 0x7fe960141010 2023-09-30T01:59:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.432+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe96011b2a0 con 0x7fe960141010 2023-09-30T01:59:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.432+0000 7fe955ffb700 1 -- 172.21.15.151:0/3536566703 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe95c052560 con 0x7fe960141010 2023-09-30T01:59:58.437 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T01:59:58.437 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:58.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 msgr2=0x7fe944062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 0x7fe944062a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe958007fd0 tx=0x7fe958009040 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 msgr2=0x7fe96014ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009290 tx=0x7fe95c016000 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 shutdown_connections 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe944060550 0x7fe944062a00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe960141010 0x7fe96014ff70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe96013dca0 0x7fe9601429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 --2- 172.21.15.151:0/3536566703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9600b0a20 0x7fe960118ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 >> 172.21.15.151:0/3536566703 conn(0x7fe960070510 msgr2=0x7fe960067ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 shutdown_connections 2023-09-30T01:59:58.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T01:59:58.436+0000 7fe966992700 1 -- 172.21.15.151:0/3536566703 wait complete. 2023-09-30T01:59:59.999 INFO:tasks.ceph.mon.a.smithi151.stderr:2023-09-30T01:59:59.996+0000 7f0d2cbf5700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:00:01.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 Processor -- start 2023-09-30T02:00:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 -- start start 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ccc0708e0 0x7f6ccc1370c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ccc06f3f0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ccc13ae20 con 0x7f6ccc070510 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd1cea700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc06e4a0 con 0x7f6ccc0708e0 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ccc0708e0 0x7f6ccc1370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd0ce8700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6ccc070510 0x7f6ccc13a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:41606/0 (socket says 172.21.15.151:41606) 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ccc0708e0 0x7f6ccc1370c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48338/0 (socket says 172.21.15.151:48338) 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd0ce8700 1 -- 172.21.15.151:0/2497330482 learned_addr learned my addr 172.21.15.151:0/2497330482 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:01.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6cd14e9700 1 --1- 172.21.15.151:0/2497330482 >> v1:172.21.15.151:6789/0 conn(0x7f6ccc06ee90 0x7f6ccc137600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58374/0 (socket says 172.21.15.151:58374) 2023-09-30T02:00:01.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 -- 172.21.15.151:0/2497330482 >> v1:172.21.15.172:6789/0 conn(0x7f6ccc070510 legacy=0x7f6ccc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 -- 172.21.15.151:0/2497330482 >> v1:172.21.15.151:6789/0 conn(0x7f6ccc06ee90 legacy=0x7f6ccc137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 -- 172.21.15.151:0/2497330482 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0ac2d0 con 0x7f6ccc0708e0 2023-09-30T02:00:01.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccb7fe700 1 -- 172.21.15.151:0/2497330482 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090348653 0 0) 0x7f6ccc13ae20 con 0x7f6ccc070510 2023-09-30T02:00:01.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccbfff700 1 --2- 172.21.15.151:0/2497330482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1370c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80098e0 tx=0x7f6cb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befe97e27d53a013 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:01.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccb7fe700 1 -- 172.21.15.151:0/2497330482 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8009e80 con 0x7f6ccc0708e0 2023-09-30T02:00:01.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccb7fe700 1 -- 172.21.15.151:0/2497330482 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb800b860 con 0x7f6ccc0708e0 2023-09-30T02:00:01.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.568+0000 7f6ccb7fe700 1 -- 172.21.15.151:0/2497330482 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800bbe0 con 0x7f6ccc0708e0 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- 172.21.15.151:0/2497330482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc0708e0 msgr2=0x7f6ccc1370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/2497330482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1370c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80098e0 tx=0x7f6cb80091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- 172.21.15.151:0/2497330482 shutdown_connections 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/2497330482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1370c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- 172.21.15.151:0/2497330482 >> 172.21.15.151:0/2497330482 conn(0x7f6ccc0b03f0 msgr2=0x7f6ccc0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- 172.21.15.151:0/2497330482 shutdown_connections 2023-09-30T02:00:01.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- 172.21.15.151:0/2497330482 wait complete. 2023-09-30T02:00:01.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 Processor -- start 2023-09-30T02:00:01.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.572+0000 7f6cd1cea700 1 -- start start 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ccc070510 0x7f6ccc13b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0aa4d0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc13d1c0 con 0x7f6ccc0708e0 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc13f2d0 con 0x7f6ccc070510 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:01.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6ccbfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ccc070510 0x7f6ccc13b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48350/0 (socket says 172.21.15.151:48350) 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 -- 172.21.15.151:0/1577984982 learned_addr learned my addr 172.21.15.151:0/1577984982 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd14e9700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 -- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ccc070510 msgr2=0x7f6ccc13b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ccc070510 0x7f6ccc13b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 -- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 msgr2=0x7f6ccc1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 -- 172.21.15.151:0/1577984982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb8009420 con 0x7f6ccc06ee90 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd14e9700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd0ce8700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000c7e0 tx=0x7f6cc0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:01.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0005b30 con 0x7f6ccc06ee90 2023-09-30T02:00:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc0005c90 con 0x7f6ccc06ee90 2023-09-30T02:00:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc1503f0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0007300 con 0x7f6ccc06ee90 2023-09-30T02:00:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc13c440 con 0x7f6ccc06ee90 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc13b4c0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.576+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6cc00076b0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.580+0000 7f6cc8ff9700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 0x7f6cb0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.580+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6cc007bb40 con 0x7f6ccc06ee90 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.580+0000 7f6ccbfff700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 0x7f6cb0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:01.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.580+0000 7f6ccbfff700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 0x7f6cb0062a30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80093f0 tx=0x7f6cb8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:01.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.588+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ccc13b4c0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ccc0008d0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cc8ff9700 1 -- 172.21.15.151:0/1577984982 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6cc004fdb0 con 0x7f6ccc06ee90 2023-09-30T02:00:01.784 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:01.784 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 msgr2=0x7f6cb0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 0x7f6cb0062a30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80093f0 tx=0x7f6cb8017000 comp rx=0 tx=0).stop 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 msgr2=0x7f6ccc13e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000c7e0 tx=0x7f6cc0007df0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 shutdown_connections 2023-09-30T02:00:01.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6cb0060580 0x7f6cb0062a30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ccc0708e0 0x7f6ccc1352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ccc070510 0x7f6ccc13b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 --2- 172.21.15.151:0/1577984982 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ccc06ee90 0x7f6ccc13e7c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 >> 172.21.15.151:0/1577984982 conn(0x7f6ccc0b03f0 msgr2=0x7f6ccc0ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 shutdown_connections 2023-09-30T02:00:01.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:01.780+0000 7f6cd1cea700 1 -- 172.21.15.151:0/1577984982 wait complete. 2023-09-30T02:00:04.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 Processor -- start 2023-09-30T02:00:04.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 -- start start 2023-09-30T02:00:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7b640b3b90 0x7f7b6406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b640b3f60 0x7f7b6406d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6406c720 con 0x7f7b6406db80 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6406ca00 con 0x7f7b640b3b90 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6406b810 con 0x7f7b640b3f60 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7b640b3b90 0x7f7b6406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b62d9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7b6406db80 0x7f7b640ae5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58382/0 (socket says 172.21.15.151:58382) 2023-09-30T02:00:04.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b62d9d700 1 -- 172.21.15.151:0/2800131554 learned_addr learned my addr 172.21.15.151:0/2800131554 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b61d9b700 1 --2- 172.21.15.151:0/2800131554 >> v2:172.21.15.172:3300/0 conn(0x7f7b640b3f60 0x7f7b6406d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6159a700 1 -- 172.21.15.151:0/2800131554 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520533202 0 0) 0x7f7b6406c720 con 0x7f7b6406db80 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6159a700 1 -- 172.21.15.151:0/2800131554 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4c003570 con 0x7f7b6406db80 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 -- 172.21.15.151:0/2800131554 >> v2:172.21.15.172:3300/0 conn(0x7f7b640b3f60 msgr2=0x7f7b6406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 --2- 172.21.15.151:0/2800131554 >> v2:172.21.15.172:3300/0 conn(0x7f7b640b3f60 0x7f7b6406d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 -- 172.21.15.151:0/2800131554 >> v1:172.21.15.151:6789/0 conn(0x7f7b6406db80 legacy=0x7f7b640ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:04.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 -- 172.21.15.151:0/2800131554 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640b0170 con 0x7f7b640b3b90 2023-09-30T02:00:04.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6259c700 1 --2- 172.21.15.151:0/2800131554 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3b90 0x7f7b6406d100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406bde0 tx=0x7f7b580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710643f6e09c1c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6159a700 1 -- 172.21.15.151:0/2800131554 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011630 con 0x7f7b640b3b90 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6159a700 1 -- 172.21.15.151:0/2800131554 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b580117b0 con 0x7f7b640b3b90 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6159a700 1 -- 172.21.15.151:0/2800131554 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58011b70 con 0x7f7b640b3b90 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 -- 172.21.15.151:0/2800131554 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3b90 msgr2=0x7f7b6406d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.912+0000 7f7b6359e700 1 --2- 172.21.15.151:0/2800131554 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3b90 0x7f7b6406d100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406bde0 tx=0x7f7b580091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- 172.21.15.151:0/2800131554 shutdown_connections 2023-09-30T02:00:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 --2- 172.21.15.151:0/2800131554 >> v2:172.21.15.172:3300/0 conn(0x7f7b640b3f60 0x7f7b6406d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 --2- 172.21.15.151:0/2800131554 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3b90 0x7f7b6406d100 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- 172.21.15.151:0/2800131554 >> 172.21.15.151:0/2800131554 conn(0x7f7b64063990 msgr2=0x7f7b64070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:04.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- 172.21.15.151:0/2800131554 shutdown_connections 2023-09-30T02:00:04.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- 172.21.15.151:0/2800131554 wait complete. 2023-09-30T02:00:04.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 Processor -- start 2023-09-30T02:00:04.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- start start 2023-09-30T02:00:04.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b6406db80 0x7f7b640a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b640b3b90 0x7f7b640abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640ab420 con 0x7f7b640b3f60 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640ab700 con 0x7f7b640b3b90 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640aa570 con 0x7f7b6406db80 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b62d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b62d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:48372/0 (socket says 172.21.15.151:48372) 2023-09-30T02:00:04.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b62d9d700 1 -- 172.21.15.151:0/1798638840 learned_addr learned my addr 172.21.15.151:0/1798638840 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b61d9b700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b640b3b90 0x7f7b640abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.916+0000 7f7b6259c700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b6406db80 0x7f7b640a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 -- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b6406db80 msgr2=0x7f7b640a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b6406db80 0x7f7b640a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 -- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b640b3b90 msgr2=0x7f7b640abf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b640b3b90 0x7f7b640abf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 -- 172.21.15.151:0/1798638840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58009420 con 0x7f7b640b3f60 2023-09-30T02:00:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b62d9d700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00c560 tx=0x7f7b5c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:04.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c005c00 con 0x7f7b640b3f60 2023-09-30T02:00:04.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c005d60 con 0x7f7b640b3f60 2023-09-30T02:00:04.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b64142030 con 0x7f7b640b3f60 2023-09-30T02:00:04.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c007300 con 0x7f7b640b3f60 2023-09-30T02:00:04.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640acba0 con 0x7f7b640b3f60 2023-09-30T02:00:04.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.920+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b640abbd0 con 0x7f7b640b3f60 2023-09-30T02:00:04.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.924+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7b5c01b3b0 con 0x7f7b640b3f60 2023-09-30T02:00:04.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.924+0000 7f7b527fc700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 0x7f7b44062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:04.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.924+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7b5c07bbd0 con 0x7f7b640b3f60 2023-09-30T02:00:04.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.924+0000 7f7b6259c700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 0x7f7b44062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:04.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.924+0000 7f7b6259c700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 0x7f7b44062a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406bde0 tx=0x7f7b58017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:04.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:04.928+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7b5c04c320 con 0x7f7b640b3f60 2023-09-30T02:00:05.127 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.124+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b640ab560 con 0x7f7b640b3f60 2023-09-30T02:00:05.128 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.124+0000 7f7b527fc700 1 -- 172.21.15.151:0/1798638840 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b5c017100 con 0x7f7b640b3f60 2023-09-30T02:00:05.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:05.129 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 msgr2=0x7f7b44062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 0x7f7b44062a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406bde0 tx=0x7f7b58017000 comp rx=0 tx=0).stop 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 msgr2=0x7f7b640ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00c560 tx=0x7f7b5c007d90 comp rx=0 tx=0).stop 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 shutdown_connections 2023-09-30T02:00:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7b44060550 0x7f7b44062a00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b640b3f60 0x7f7b640ac440 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b640b3b90 0x7f7b640abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 --2- 172.21.15.151:0/1798638840 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b6406db80 0x7f7b640a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 >> 172.21.15.151:0/1798638840 conn(0x7f7b64063990 msgr2=0x7f7b64142570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 shutdown_connections 2023-09-30T02:00:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:05.128+0000 7f7b6359e700 1 -- 172.21.15.151:0/1798638840 wait complete. 2023-09-30T02:00:08.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:08.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 Processor -- start 2023-09-30T02:00:08.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 -- start start 2023-09-30T02:00:08.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0d5406e390 0x7f0d540ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 0x7f0d54137f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.257 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d540abcf0 con 0x7f0d540ab7b0 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5406e760 con 0x7f0d5406e390 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a993700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5413f2f0 con 0x7f0d5413ee80 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a192700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 0x7f0d54137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d59190700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0d540ab7b0 0x7f0d5413e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58392/0 (socket says 172.21.15.151:58392) 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d59190700 1 -- 172.21.15.151:0/2924186826 learned_addr learned my addr 172.21.15.151:0/2924186826 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:08.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d59991700 1 --2- 172.21.15.151:0/2924186826 >> v2:172.21.15.172:3300/0 conn(0x7f0d5406e390 0x7f0d540ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5898f700 1 -- 172.21.15.151:0/2924186826 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011122698 0 0) 0x7f0d540abcf0 con 0x7f0d540ab7b0 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5898f700 1 -- 172.21.15.151:0/2924186826 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40003570 con 0x7f0d540ab7b0 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a192700 1 -- 172.21.15.151:0/2924186826 >> v2:172.21.15.172:3300/0 conn(0x7f0d5406e390 msgr2=0x7f0d540ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a192700 1 --2- 172.21.15.151:0/2924186826 >> v2:172.21.15.172:3300/0 conn(0x7f0d5406e390 0x7f0d540ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a192700 1 -- 172.21.15.151:0/2924186826 >> v1:172.21.15.151:6789/0 conn(0x7f0d540ab7b0 legacy=0x7f0d5413e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.252+0000 7f0d5a192700 1 -- 172.21.15.151:0/2924186826 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540ae230 con 0x7f0d5413ee80 2023-09-30T02:00:08.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a192700 1 --2- 172.21.15.151:0/2924186826 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 0x7f0d54137f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009e60 tx=0x7f0d4c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea898bd22e89429 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:08.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5898f700 1 -- 172.21.15.151:0/2924186826 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00e070 con 0x7f0d5413ee80 2023-09-30T02:00:08.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5898f700 1 -- 172.21.15.151:0/2924186826 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c00bab0 con 0x7f0d5413ee80 2023-09-30T02:00:08.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5898f700 1 -- 172.21.15.151:0/2924186826 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00be70 con 0x7f0d5413ee80 2023-09-30T02:00:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- 172.21.15.151:0/2924186826 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 msgr2=0x7f0d54137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 --2- 172.21.15.151:0/2924186826 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 0x7f0d54137f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009e60 tx=0x7f0d4c009780 comp rx=0 tx=0).stop 2023-09-30T02:00:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- 172.21.15.151:0/2924186826 shutdown_connections 2023-09-30T02:00:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 --2- 172.21.15.151:0/2924186826 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5413ee80 0x7f0d54137f60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 --2- 172.21.15.151:0/2924186826 >> v2:172.21.15.172:3300/0 conn(0x7f0d5406e390 0x7f0d540ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- 172.21.15.151:0/2924186826 >> 172.21.15.151:0/2924186826 conn(0x7f0d5406ae50 msgr2=0x7f0d54070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:08.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- 172.21.15.151:0/2924186826 shutdown_connections 2023-09-30T02:00:08.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- 172.21.15.151:0/2924186826 wait complete. 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 Processor -- start 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.256+0000 7f0d5a993700 1 -- start start 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d540ab7b0 0x7f0d5406c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d5413ee80 0x7f0d5406c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540a6cc0 con 0x7f0d5413ee80 2023-09-30T02:00:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540a6f30 con 0x7f0d540ab7b0 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540a5c50 con 0x7f0d5406e390 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:46056/0 (socket says 172.21.15.151:46056) 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 -- 172.21.15.151:0/659257997 learned_addr learned my addr 172.21.15.151:0/659257997 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a192700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d5413ee80 0x7f0d5406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59190700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d540ab7b0 0x7f0d5406c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 -- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d540ab7b0 msgr2=0x7f0d5406c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d540ab7b0 0x7f0d5406c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 -- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d5413ee80 msgr2=0x7f0d5406c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d5413ee80 0x7f0d5406c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 -- 172.21.15.151:0/659257997 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009a20 con 0x7f0d5406e390 2023-09-30T02:00:08.265 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d59991700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000ecb0 tx=0x7f0d500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5000c850 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5000c9b0 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d540a5ef0 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5000cd70 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d540a4bb0 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.260+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d540a4e00 con 0x7f0d5406e390 2023-09-30T02:00:08.266 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.264+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0d50013680 con 0x7f0d5406e390 2023-09-30T02:00:08.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.264+0000 7f0d49ffb700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 0x7f0d38062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:08.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.264+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0d5007c850 con 0x7f0d5406e390 2023-09-30T02:00:08.267 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.264+0000 7f0d59190700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 0x7f0d38062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:08.268 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.264+0000 7f0d59190700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 0x7f0d38062a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d540a5520 tx=0x7f0d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:08.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.272+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d5004cf90 con 0x7f0d5406e390 2023-09-30T02:00:08.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.464+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0d540008d0 con 0x7f0d5406e390 2023-09-30T02:00:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d49ffb700 1 -- 172.21.15.151:0/659257997 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0d500505b0 con 0x7f0d5406e390 2023-09-30T02:00:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:08.471 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 msgr2=0x7f0d38062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 0x7f0d38062a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d540a5520 tx=0x7f0d44009040 comp rx=0 tx=0).stop 2023-09-30T02:00:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 msgr2=0x7f0d540a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000ecb0 tx=0x7f0d500091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 shutdown_connections 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0d38060550 0x7f0d38062a00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0d5413ee80 0x7f0d5406c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d540ab7b0 0x7f0d5406c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 --2- 172.21.15.151:0/659257997 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0d5406e390 0x7f0d540a5710 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 >> 172.21.15.151:0/659257997 conn(0x7f0d5406ae50 msgr2=0x7f0d540a80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 shutdown_connections 2023-09-30T02:00:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:08.468+0000 7f0d5a993700 1 -- 172.21.15.151:0/659257997 wait complete. 2023-09-30T02:00:11.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:11.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.592+0000 7f513b59e700 1 Processor -- start 2023-09-30T02:00:11.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- start start 2023-09-30T02:00:11.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f513c141220 0x7f513c0a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 0x7f513c13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c0aa370 con 0x7f513c0675a0 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c067970 con 0x7f513c141220 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c141680 con 0x7f513c13f780 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f513c141220 0x7f513c0a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 0x7f513c13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f513c141220 0x7f513c0a9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44258/0 (socket says 172.21.15.151:44258) 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 0x7f513c13fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56946/0 (socket says 172.21.15.151:56946) 2023-09-30T02:00:11.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 -- 172.21.15.151:0/3840045462 learned_addr learned my addr 172.21.15.151:0/3840045462 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 -- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 msgr2=0x7f513c13fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 --2- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 0x7f513c13fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 -- 172.21.15.151:0/3840045462 >> v1:172.21.15.172:6789/0 conn(0x7f513c0675a0 legacy=0x7f513c140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 -- 172.21.15.151:0/3840045462 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c0aa0b0 con 0x7f513c141220 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5131d9b700 1 --2- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c141220 0x7f513c0a9b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51240099b0 tx=0x7f5124009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da78a8cce559a6c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5139d9b700 1 -- 172.21.15.151:0/3840045462 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124009ea0 con 0x7f513c141220 2023-09-30T02:00:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5139d9b700 1 -- 172.21.15.151:0/3840045462 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512400b8f0 con 0x7f513c141220 2023-09-30T02:00:11.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f5139d9b700 1 -- 172.21.15.151:0/3840045462 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400bc90 con 0x7f513c141220 2023-09-30T02:00:11.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c141220 msgr2=0x7f513c0a9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 --2- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c141220 0x7f513c0a9b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51240099b0 tx=0x7f5124009250 comp rx=0 tx=0).stop 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- 172.21.15.151:0/3840045462 shutdown_connections 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 --2- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c13f780 0x7f513c13fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 --2- 172.21.15.151:0/3840045462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c141220 0x7f513c0a9b70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- 172.21.15.151:0/3840045462 >> 172.21.15.151:0/3840045462 conn(0x7f513c0b08c0 msgr2=0x7f513c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- 172.21.15.151:0/3840045462 shutdown_connections 2023-09-30T02:00:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.596+0000 7f513b59e700 1 -- 172.21.15.151:0/3840045462 wait complete. 2023-09-30T02:00:11.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 Processor -- start 2023-09-30T02:00:11.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 -- start start 2023-09-30T02:00:11.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 0x7f513c13e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c141220 0x7f513c1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c13e180 con 0x7f513c13f780 2023-09-30T02:00:11.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c118a40 con 0x7f513c0675a0 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c13e3f0 con 0x7f513c141220 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44266/0 (socket says 172.21.15.151:44266) 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 -- 172.21.15.151:0/2628531569 learned_addr learned my addr 172.21.15.151:0/2628531569 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513ad9d700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c141220 0x7f513c1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513a59c700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 0x7f513c13e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 -- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c141220 msgr2=0x7f513c1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c141220 0x7f513c1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 -- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 msgr2=0x7f513c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 0x7f513c13e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f5131d9b700 1 -- 172.21.15.151:0/2628531569 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51240094f0 con 0x7f513c13f780 2023-09-30T02:00:11.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.600+0000 7f513a59c700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 0x7f513c13e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:11.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5131d9b700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f513c06b080 tx=0x7f5124017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:11.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400bc70 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124025440 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c13b520 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124025800 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c13dce0 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c0008d0 con 0x7f513c13f780 2023-09-30T02:00:11.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f512402e440 con 0x7f513c13f780 2023-09-30T02:00:11.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 0x7f511c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:11.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f512407e2d0 con 0x7f513c13f780 2023-09-30T02:00:11.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.604+0000 7f513a59c700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 0x7f511c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:11.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.608+0000 7f513a59c700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 0x7f511c062a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f512c007fd0 tx=0x7f512c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:11.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.612+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f512404eb30 con 0x7f513c13f780 2023-09-30T02:00:11.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.808+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f513c0a8eb0 con 0x7f513c13f780 2023-09-30T02:00:11.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.808+0000 7f5132ffd700 1 -- 172.21.15.151:0/2628531569 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5124052150 con 0x7f513c13f780 2023-09-30T02:00:11.814 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:11.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 msgr2=0x7f511c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 0x7f511c062a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f512c007fd0 tx=0x7f512c009040 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 msgr2=0x7f513c06d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f513c06b080 tx=0x7f5124017000 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 shutdown_connections 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f511c060550 0x7f511c062a00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f513c141220 0x7f513c1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f513c13f780 0x7f513c06d6e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 --2- 172.21.15.151:0/2628531569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f513c0675a0 0x7f513c13e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 >> 172.21.15.151:0/2628531569 conn(0x7f513c0b08c0 msgr2=0x7f513c0a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 shutdown_connections 2023-09-30T02:00:11.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:11.812+0000 7f513b59e700 1 -- 172.21.15.151:0/2628531569 wait complete. 2023-09-30T02:00:14.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:14.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 Processor -- start 2023-09-30T02:00:14.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 -- start start 2023-09-30T02:00:14.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 0x7f362c06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c138c00 con 0x7f362c13afd0 2023-09-30T02:00:14.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362c0abbd0 con 0x7f362c0ac510 2023-09-30T02:00:14.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632f26700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0abdd0 con 0x7f362c0ac140 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632725700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f362c13afd0 0x7f362c0a8b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:54388/0 (socket says 172.21.15.151:54388) 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3632725700 1 -- 172.21.15.151:0/2614961514 learned_addr learned my addr 172.21.15.151:0/2614961514 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631723700 1 --2- 172.21.15.151:0/2614961514 >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 0x7f362c06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3630f22700 1 -- 172.21.15.151:0/2614961514 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706364187 0 0) 0x7f362c138c00 con 0x7f362c13afd0 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 -- 172.21.15.151:0/2614961514 >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 msgr2=0x7f362c06e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 --2- 172.21.15.151:0/2614961514 >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 0x7f362c06e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 -- 172.21.15.151:0/2614961514 >> v1:172.21.15.151:6789/0 conn(0x7f362c13afd0 legacy=0x7f362c0a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 -- 172.21.15.151:0/2614961514 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c0a9b50 con 0x7f362c0ac140 2023-09-30T02:00:14.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631723700 1 --2- 172.21.15.151:0/2614961514 >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 0x7f362c06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:14.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3631f24700 1 --2- 172.21.15.151:0/2614961514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0aa110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f36200098e0 tx=0x7f36200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1598094f0addb1e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:14.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3630f22700 1 -- 172.21.15.151:0/2614961514 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620009e80 con 0x7f362c0ac140 2023-09-30T02:00:14.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3630f22700 1 -- 172.21.15.151:0/2614961514 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362000b860 con 0x7f362c0ac140 2023-09-30T02:00:14.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.936+0000 7f3630f22700 1 -- 172.21.15.151:0/2614961514 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000bbe0 con 0x7f362c0ac140 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2614961514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 msgr2=0x7f362c0aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- 172.21.15.151:0/2614961514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0aa110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f36200098e0 tx=0x7f36200091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2614961514 shutdown_connections 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- 172.21.15.151:0/2614961514 >> v2:172.21.15.172:3300/0 conn(0x7f362c0ac510 0x7f362c06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- 172.21.15.151:0/2614961514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0aa110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2614961514 >> 172.21.15.151:0/2614961514 conn(0x7f362c070510 msgr2=0x7f362c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:14.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2614961514 shutdown_connections 2023-09-30T02:00:14.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2614961514 wait complete. 2023-09-30T02:00:14.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 Processor -- start 2023-09-30T02:00:14.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- start start 2023-09-30T02:00:14.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f362c0ac510 0x7f362c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f362c13afd0 0x7f362c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0a4c60 con 0x7f362c0ac510 2023-09-30T02:00:14.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0a6030 con 0x7f362c13afd0 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0a50f0 con 0x7f362c0ac140 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56980/0 (socket says 172.21.15.151:56980) 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631723700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f362c0ac510 0x7f362c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 -- 172.21.15.151:0/2198062913 learned_addr learned my addr 172.21.15.151:0/2198062913 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632725700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f362c13afd0 0x7f362c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 -- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f362c13afd0 msgr2=0x7f362c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:14.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f362c13afd0 0x7f362c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 -- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f362c0ac510 msgr2=0x7f362c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:14.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f362c0ac510 0x7f362c06ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:14.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 -- 172.21.15.151:0/2198062913 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620009420 con 0x7f362c0ac140 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3631f24700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f362c0abf10 tx=0x7f3620017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000bbe0 con 0x7f362c0ac140 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c13e870 con 0x7f362c0ac140 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3620000c90 con 0x7f362c0ac140 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620011670 con 0x7f362c0ac140 2023-09-30T02:00:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.940+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c150840 con 0x7f362c0ac140 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c0a61f0 con 0x7f362c0ac140 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3620011aa0 con 0x7f362c0ac140 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3629ffb700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 0x7f3618062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f362007dfc0 con 0x7f362c0ac140 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3631723700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 0x7f3618062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:14.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.944+0000 7f3631723700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 0x7f3618062a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362c13a0a0 tx=0x7f361c0099d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:14.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:14.956+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f362004e700 con 0x7f362c0ac140 2023-09-30T02:00:15.155 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.152+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f362c150b20 con 0x7f362c0ac140 2023-09-30T02:00:15.156 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.152+0000 7f3629ffb700 1 -- 172.21.15.151:0/2198062913 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3620051d20 con 0x7f362c0ac140 2023-09-30T02:00:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:15.156 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 msgr2=0x7f3618062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 0x7f3618062a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362c13a0a0 tx=0x7f361c0099d0 comp rx=0 tx=0).stop 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 msgr2=0x7f362c0a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f362c0abf10 tx=0x7f3620017000 comp rx=0 tx=0).stop 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 shutdown_connections 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3618060550 0x7f3618062a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f362c13afd0 0x7f362c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f362c0ac510 0x7f362c06ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 --2- 172.21.15.151:0/2198062913 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f362c0ac140 0x7f362c0a4570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:15.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 >> 172.21.15.151:0/2198062913 conn(0x7f362c070510 msgr2=0x7f362c0708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:15.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 shutdown_connections 2023-09-30T02:00:15.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:15.156+0000 7f3632f26700 1 -- 172.21.15.151:0/2198062913 wait complete. 2023-09-30T02:00:18.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:18.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 Processor -- start 2023-09-30T02:00:18.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 -- start start 2023-09-30T02:00:18.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8bc406a450 0x7f8bc406c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 0x7f8bc40aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc406ca40 con 0x7f8bc406a080 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc406e360 con 0x7f8bc406a450 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc406e070 con 0x7f8bc4139f10 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 0x7f8bc40aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44288/0 (socket says 172.21.15.151:44288) 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc259c700 1 -- 172.21.15.151:0/1042930999 learned_addr learned my addr 172.21.15.151:0/1042930999 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:18.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bb9d9b700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.172:3300/0 conn(0x7f8bc406a450 0x7f8bc406c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc2d9d700 1 -- 172.21.15.151:0/1042930999 >> v2:172.21.15.172:3300/0 conn(0x7f8bc406a450 msgr2=0x7f8bc406c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc2d9d700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.172:3300/0 conn(0x7f8bc406a450 0x7f8bc406c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc2d9d700 1 -- 172.21.15.151:0/1042930999 >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 msgr2=0x7f8bc406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.276+0000 7f8bc2d9d700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc2d9d700 1 -- 172.21.15.151:0/1042930999 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc40acf40 con 0x7f8bc4139f10 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:18.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc2d9d700 1 --2- 172.21.15.151:0/1042930999 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 0x7f8bc40aa5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8bc406a950 tx=0x7f8bbc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=240be64502468738 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc1d9b700 1 -- 172.21.15.151:0/1042930999 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00e070 con 0x7f8bc4139f10 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc1d9b700 1 -- 172.21.15.151:0/1042930999 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc00bab0 con 0x7f8bc4139f10 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc1d9b700 1 -- 172.21.15.151:0/1042930999 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00be70 con 0x7f8bc4139f10 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- 172.21.15.151:0/1042930999 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 msgr2=0x7f8bc40aa5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1042930999 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 0x7f8bc40aa5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8bc406a950 tx=0x7f8bbc009780 comp rx=0 tx=0).stop 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- 172.21.15.151:0/1042930999 shutdown_connections 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1042930999 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc4139f10 0x7f8bc40aa5a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.172:3300/0 conn(0x7f8bc406a450 0x7f8bc406c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1042930999 >> v2:172.21.15.151:3300/0 conn(0x7f8bc406a080 0x7f8bc406e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- 172.21.15.151:0/1042930999 >> 172.21.15.151:0/1042930999 conn(0x7f8bc40b0a40 msgr2=0x7f8bc40b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- 172.21.15.151:0/1042930999 shutdown_connections 2023-09-30T02:00:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- 172.21.15.151:0/1042930999 wait complete. 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 Processor -- start 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.280+0000 7f8bc359e700 1 -- start start 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 0x7f8bc414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc413e1e0 con 0x7f8bc4139f10 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41372b0 con 0x7f8bc406a080 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc413c450 con 0x7f8bc406a450 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 0x7f8bc414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44298/0 (socket says 172.21.15.151:44298) 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc2d9d700 1 -- 172.21.15.151:0/1322799644 learned_addr learned my addr 172.21.15.151:0/1322799644 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc259c700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 msgr2=0x7f8bc40af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc259c700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 msgr2=0x7f8bc40af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:00:18.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:00:18.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 msgr2=0x7f8bc40af540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T02:00:18.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 msgr2=0x7f8bc414d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:18.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 -- 172.21.15.151:0/1322799644 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb40094f0 con 0x7f8bc406a450 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc2d9d700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bb9d9b700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 0x7f8bc414cdb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00e9d0 tx=0x7f8bac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac00eec0 con 0x7f8bc406a450 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac00c850 con 0x7f8bc406a450 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bbc009a20 con 0x7f8bc406a450 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac00cbd0 con 0x7f8bc406a450 2023-09-30T02:00:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.284+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc4136540 con 0x7f8bc406a450 2023-09-30T02:00:18.291 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc404be00 con 0x7f8bc406a450 2023-09-30T02:00:18.291 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8bac013470 con 0x7f8bc406a450 2023-09-30T02:00:18.291 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bbaffd700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 0x7f8ba4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:18.292 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8bac07cbb0 con 0x7f8bc406a450 2023-09-30T02:00:18.292 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 0x7f8ba4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:18.293 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.288+0000 7f8bc259c700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 0x7f8ba4062a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006d40 tx=0x7f8bb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:18.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.296+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8bac04d2f0 con 0x7f8bc406a450 2023-09-30T02:00:18.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.492+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8bc41373f0 con 0x7f8bc406a450 2023-09-30T02:00:18.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.492+0000 7f8bbaffd700 1 -- 172.21.15.151:0/1322799644 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8bac050910 con 0x7f8bc406a450 2023-09-30T02:00:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:18.497 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:18.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 msgr2=0x7f8ba4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:18.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 0x7f8ba4062a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006d40 tx=0x7f8bb4008040 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 msgr2=0x7f8bc414cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 0x7f8bc414cdb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00e9d0 tx=0x7f8bac0091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 shutdown_connections 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ba4060550 0x7f8ba4062a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bc4139f10 0x7f8bc414d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bc406a450 0x7f8bc414cdb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 --2- 172.21.15.151:0/1322799644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bc406a080 0x7f8bc40af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 >> 172.21.15.151:0/1322799644 conn(0x7f8bc40b0a40 msgr2=0x7f8bc41392b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 shutdown_connections 2023-09-30T02:00:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:18.496+0000 7f8bc359e700 1 -- 172.21.15.151:0/1322799644 wait complete. 2023-09-30T02:00:21.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:21.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.628+0000 7f8485633700 1 Processor -- start 2023-09-30T02:00:21.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.628+0000 7f8485633700 1 -- start start 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f848006b760 0x7f848013fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f848013ffa0 0x7f84801403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848006bf90 con 0x7f84800b4510 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8480140930 con 0x7f848013ffa0 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8480141a50 con 0x7f848006b760 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f848013ffa0 0x7f84801403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f848013ffa0 0x7f84801403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35418/0 (socket says 172.21.15.151:35418) 2023-09-30T02:00:21.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 -- 172.21.15.151:0/3797643998 learned_addr learned my addr 172.21.15.151:0/3797643998 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:21.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f847f7fe700 1 --2- 172.21.15.151:0/3797643998 >> v2:172.21.15.172:3300/0 conn(0x7f848006b760 0x7f848013fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 -- 172.21.15.151:0/3797643998 >> v2:172.21.15.172:3300/0 conn(0x7f848006b760 msgr2=0x7f848013fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 --2- 172.21.15.151:0/3797643998 >> v2:172.21.15.172:3300/0 conn(0x7f848006b760 0x7f848013fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f847effd700 1 -- 172.21.15.151:0/3797643998 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578289486 0 0) 0x7f848006bf90 con 0x7f84800b4510 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 -- 172.21.15.151:0/3797643998 >> v1:172.21.15.172:6789/0 conn(0x7f84800b4510 legacy=0x7f848006b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 -- 172.21.15.151:0/3797643998 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84800b48e0 con 0x7f848013ffa0 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8484e32700 1 --2- 172.21.15.151:0/3797643998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f848013ffa0 0x7f84801403f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8480064530 tx=0x7f8470009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb4d7ce56b56c4a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f847effd700 1 -- 172.21.15.151:0/3797643998 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470009ea0 con 0x7f848013ffa0 2023-09-30T02:00:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f847effd700 1 -- 172.21.15.151:0/3797643998 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847000b820 con 0x7f848013ffa0 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f847effd700 1 -- 172.21.15.151:0/3797643998 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847000bbc0 con 0x7f848013ffa0 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- 172.21.15.151:0/3797643998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f848013ffa0 msgr2=0x7f84801403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 --2- 172.21.15.151:0/3797643998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f848013ffa0 0x7f84801403f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8480064530 tx=0x7f8470009250 comp rx=0 tx=0).stop 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- 172.21.15.151:0/3797643998 shutdown_connections 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 --2- 172.21.15.151:0/3797643998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f848013ffa0 0x7f84801403f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 --2- 172.21.15.151:0/3797643998 >> v2:172.21.15.172:3300/0 conn(0x7f848006b760 0x7f848013fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- 172.21.15.151:0/3797643998 >> 172.21.15.151:0/3797643998 conn(0x7f8480070f90 msgr2=0x7f84800b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- 172.21.15.151:0/3797643998 shutdown_connections 2023-09-30T02:00:21.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.632+0000 7f8485633700 1 -- 172.21.15.151:0/3797643998 wait complete. 2023-09-30T02:00:21.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 Processor -- start 2023-09-30T02:00:21.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 -- start start 2023-09-30T02:00:21.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 0x7f848011bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 0x7f848013acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84800ad2f0 0x7f848013b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848011d9c0 con 0x7f84800b4510 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84800aa860 con 0x7f848013ffa0 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8485633700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848013b730 con 0x7f84800ad2f0 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 0x7f848011bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8484e32700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84800ad2f0 0x7f848013b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 0x7f848011bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35434/0 (socket says 172.21.15.151:35434) 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847ffff700 1 -- 172.21.15.151:0/3534718451 learned_addr learned my addr 172.21.15.151:0/3534718451 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 0x7f848013acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 -- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84800ad2f0 msgr2=0x7f848013b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84800ad2f0 0x7f848013b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 -- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 msgr2=0x7f848011bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 0x7f848011bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 -- 172.21.15.151:0/3534718451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84700094f0 con 0x7f848013ffa0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f847f7fe700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 0x7f848013acb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f847400ea80 tx=0x7f84740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.636+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847400eec0 con 0x7f848013ffa0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847400c7f0 con 0x7f848013ffa0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84800ac6b0 con 0x7f848013ffa0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847400cb70 con 0x7f848013ffa0 2023-09-30T02:00:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84800aacf0 con 0x7f848013ffa0 2023-09-30T02:00:21.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84800670a0 con 0x7f848013ffa0 2023-09-30T02:00:21.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8474005a00 con 0x7f848013ffa0 2023-09-30T02:00:21.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8467fff700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 0x7f846c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:21.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f847407cc00 con 0x7f848013ffa0 2023-09-30T02:00:21.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.640+0000 7f847ffff700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 0x7f846c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:21.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.644+0000 7f847ffff700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 0x7f846c062a30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8478001560 tx=0x7f84780017a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:21.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.648+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f847404d3c0 con 0x7f848013ffa0 2023-09-30T02:00:21.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.840+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f84800678e0 con 0x7f848013ffa0 2023-09-30T02:00:21.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8467fff700 1 -- 172.21.15.151:0/3534718451 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8474021100 con 0x7f848013ffa0 2023-09-30T02:00:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:21.846 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 msgr2=0x7f846c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 0x7f846c062a30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8478001560 tx=0x7f84780017a0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 msgr2=0x7f848013acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 0x7f848013acb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f847400ea80 tx=0x7f84740091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 shutdown_connections 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f846c060580 0x7f846c062a30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84800ad2f0 0x7f848013b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f848013ffa0 0x7f848013acb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 --2- 172.21.15.151:0/3534718451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84800b4510 0x7f848011bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:21.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 >> 172.21.15.151:0/3534718451 conn(0x7f8480070f90 msgr2=0x7f84800b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:21.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 shutdown_connections 2023-09-30T02:00:21.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:21.844+0000 7f8485633700 1 -- 172.21.15.151:0/3534718451 wait complete. 2023-09-30T02:00:24.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:24.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 Processor -- start 2023-09-30T02:00:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 -- start start 2023-09-30T02:00:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f25400accd0 0x7f254013d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254013cc00 con 0x7f254013d5d0 2023-09-30T02:00:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400ad0a0 con 0x7f25400aa820 2023-09-30T02:00:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2546cd9700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254006b590 con 0x7f25400accd0 2023-09-30T02:00:25.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f25400accd0 0x7f254013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f25454d6700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f254013d5d0 0x7f25400aa110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44368/0 (socket says 172.21.15.151:44368) 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f25400accd0 0x7f254013d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35448/0 (socket says 172.21.15.151:35448) 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f25454d6700 1 -- 172.21.15.151:0/2523833674 learned_addr learned my addr 172.21.15.151:0/2523833674 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 -- 172.21.15.151:0/2523833674 >> v1:172.21.15.172:6789/0 conn(0x7f25400aa820 legacy=0x7f2540143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 -- 172.21.15.151:0/2523833674 >> v1:172.21.15.151:6789/0 conn(0x7f254013d5d0 legacy=0x7f25400aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 -- 172.21.15.151:0/2523833674 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254006cb60 con 0x7f25400accd0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2545cd7700 1 --2- 172.21.15.151:0/2523833674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013d090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2530009910 tx=0x7f25300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f09383f307428f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2544cd5700 1 -- 172.21.15.151:0/2523833674 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000b820 con 0x7f25400accd0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2544cd5700 1 -- 172.21.15.151:0/2523833674 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253000b980 con 0x7f25400accd0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.972+0000 7f2544cd5700 1 -- 172.21.15.151:0/2523833674 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000bd20 con 0x7f25400accd0 2023-09-30T02:00:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- 172.21.15.151:0/2523833674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 msgr2=0x7f254013d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 --2- 172.21.15.151:0/2523833674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013d090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2530009910 tx=0x7f25300091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- 172.21.15.151:0/2523833674 shutdown_connections 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 --2- 172.21.15.151:0/2523833674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013d090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- 172.21.15.151:0/2523833674 >> 172.21.15.151:0/2523833674 conn(0x7f2540070510 msgr2=0x7f2540070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- 172.21.15.151:0/2523833674 shutdown_connections 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- 172.21.15.151:0/2523833674 wait complete. 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 Processor -- start 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- start start 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254013d5d0 0x7f254014ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400a7130 con 0x7f25400accd0 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254013f4d0 con 0x7f254013d5d0 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2546cd9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400a9020 con 0x7f25400aa820 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f25454d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52690/0 (socket says 172.21.15.151:52690) 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f25454d6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:35462/0 (socket says 172.21.15.151:35462) 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 -- 172.21.15.151:0/3931978028 learned_addr learned my addr 172.21.15.151:0/3931978028 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f25464d8700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254013d5d0 0x7f254014ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 -- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254013d5d0 msgr2=0x7f254014ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254013d5d0 0x7f254014ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 -- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 msgr2=0x7f254013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 -- 172.21.15.151:0/3931978028 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530009420 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f25454d6700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.976+0000 7f2545cd7700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25400a9d90 tx=0x7f2530017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000bd00 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2530024440 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25300247c0 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254013dd20 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254014d5b0 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254014cb10 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.980+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f253002d440 con 0x7f25400aa820 2023-09-30T02:00:25.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.984+0000 7f2535ffb700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 0x7f2524062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:25.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.984+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f253007dfa0 con 0x7f25400aa820 2023-09-30T02:00:25.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.984+0000 7f25454d6700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 0x7f2524062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:25.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.984+0000 7f25454d6700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 0x7f2524062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2540144ce0 tx=0x7f253c007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:25.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:24.992+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f25300adbe0 con 0x7f25400aa820 2023-09-30T02:00:25.193 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.188+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25400008d0 con 0x7f25400aa820 2023-09-30T02:00:25.194 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.188+0000 7f2535ffb700 1 -- 172.21.15.151:0/3931978028 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2530015100 con 0x7f25400aa820 2023-09-30T02:00:25.194 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:25.194 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:25.196 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 msgr2=0x7f2524062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 0x7f2524062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2540144ce0 tx=0x7f253c007f10 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 msgr2=0x7f254013ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25400a9d90 tx=0x7f2530017000 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 shutdown_connections 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2524060550 0x7f2524062a00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f254013d5d0 0x7f254014ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f25400accd0 0x7f254013ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 --2- 172.21.15.151:0/3931978028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25400aa820 0x7f254013ea50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 >> 172.21.15.151:0/3931978028 conn(0x7f2540070510 msgr2=0x7f25400a9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 shutdown_connections 2023-09-30T02:00:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:25.192+0000 7f2546cd9700 1 -- 172.21.15.151:0/3931978028 wait complete. 2023-09-30T02:00:28.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:28.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 Processor -- start 2023-09-30T02:00:28.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 -- start start 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa11c0ab220 0x7fa11c0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c139ed0 con 0x7fa11c13d7b0 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0a9b00 con 0x7fa11c0ab5f0 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11b59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c0ad0e0 con 0x7fa11c0ab220 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11ad9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa11c13d7b0 0x7fa11c13dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44376/0 (socket says 172.21.15.151:44376) 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11ad9d700 1 -- 172.21.15.151:0/97117599 learned_addr learned my addr 172.21.15.151:0/97117599 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.316+0000 7fa11a59c700 1 --2- 172.21.15.151:0/97117599 >> v2:172.21.15.172:3300/0 conn(0x7fa11c0ab220 0x7fa11c0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11a59c700 1 -- 172.21.15.151:0/97117599 >> v1:172.21.15.172:6789/0 conn(0x7fa11c0ab5f0 legacy=0x7fa11c13a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11a59c700 1 -- 172.21.15.151:0/97117599 >> v1:172.21.15.151:6789/0 conn(0x7fa11c13d7b0 legacy=0x7fa11c13dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11a59c700 1 -- 172.21.15.151:0/97117599 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0acbf0 con 0x7fa11c0ab220 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11a59c700 1 --2- 172.21.15.151:0/97117599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab220 0x7fa11c0ac6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca07a6575057f9de server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11959a700 1 -- 172.21.15.151:0/97117599 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11400e070 con 0x7fa11c0ab220 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11959a700 1 -- 172.21.15.151:0/97117599 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11400bc30 con 0x7fa11c0ab220 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11959a700 1 -- 172.21.15.151:0/97117599 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114012430 con 0x7fa11c0ab220 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- 172.21.15.151:0/97117599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab220 msgr2=0x7fa11c0ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 --2- 172.21.15.151:0/97117599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab220 0x7fa11c0ac6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114009780 comp rx=0 tx=0).stop 2023-09-30T02:00:28.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- 172.21.15.151:0/97117599 shutdown_connections 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 --2- 172.21.15.151:0/97117599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab220 0x7fa11c0ac6b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- 172.21.15.151:0/97117599 >> 172.21.15.151:0/97117599 conn(0x7fa11c070510 msgr2=0x7fa11c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- 172.21.15.151:0/97117599 shutdown_connections 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- 172.21.15.151:0/97117599 wait complete. 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 Processor -- start 2023-09-30T02:00:28.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.320+0000 7fa11b59e700 1 -- start start 2023-09-30T02:00:28.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa11c0ab220 0x7fa11c0a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab5f0 0x7fa11c12c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0a5a10 con 0x7fa11c0ab220 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0b03f0 con 0x7fa11c0ab5f0 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c12d320 con 0x7fa11c13d7b0 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11a59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa11c0ab220 0x7fa11c0a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52696/0 (socket says 172.21.15.151:52696) 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 -- 172.21.15.151:0/2372386454 learned_addr learned my addr 172.21.15.151:0/2372386454 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:28.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa119d9b700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab5f0 0x7fa11c12c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:28.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 -- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab5f0 msgr2=0x7fa11c12c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab5f0 0x7fa11c12c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 -- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa11c0ab220 msgr2=0x7fa11c0a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa11c0ab220 0x7fa11c0a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 -- 172.21.15.151:0/2372386454 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114009a20 con 0x7fa11c13d7b0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11ad9d700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c590 tx=0x7fa10c007ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c005b30 con 0x7fa11c13d7b0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c005cb0 con 0x7fa11c13d7b0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c007320 con 0x7fa11c13d7b0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c0a5c40 con 0x7fa11c13d7b0 2023-09-30T02:00:28.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.324+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c0af900 con 0x7fa11c13d7b0 2023-09-30T02:00:28.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c0008d0 con 0x7fa11c13d7b0 2023-09-30T02:00:28.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa10c01b3b0 con 0x7fa11c13d7b0 2023-09-30T02:00:28.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa10a7fc700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 0x7fa0fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:28.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fa10c07ba70 con 0x7fa11c13d7b0 2023-09-30T02:00:28.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa11a59c700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 0x7fa0fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:28.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.328+0000 7fa11a59c700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 0x7fa0fc062a00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:28.338 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.336+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa10c04c1b0 con 0x7fa11c13d7b0 2023-09-30T02:00:28.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.536+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa11c13ad70 con 0x7fa11c13d7b0 2023-09-30T02:00:28.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.536+0000 7fa10a7fc700 1 -- 172.21.15.151:0/2372386454 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa10c017100 con 0x7fa11c13d7b0 2023-09-30T02:00:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:28.542 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:28.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 msgr2=0x7fa0fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 0x7fa0fc062a00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa114009e60 tx=0x7fa114019000 comp rx=0 tx=0).stop 2023-09-30T02:00:28.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 msgr2=0x7fa11c12cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c590 tx=0x7fa10c007ef0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 shutdown_connections 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa0fc060550 0x7fa0fc062a00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa11c13d7b0 0x7fa11c12cde0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa11c0ab5f0 0x7fa11c12c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 --2- 172.21.15.151:0/2372386454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa11c0ab220 0x7fa11c0a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 >> 172.21.15.151:0/2372386454 conn(0x7fa11c070510 msgr2=0x7fa11c13dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 shutdown_connections 2023-09-30T02:00:28.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:28.540+0000 7fa11b59e700 1 -- 172.21.15.151:0/2372386454 wait complete. 2023-09-30T02:00:31.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 Processor -- start 2023-09-30T02:00:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 -- start start 2023-09-30T02:00:31.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc06c8e0 con 0x7fc6fc06b180 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc0abe70 con 0x7fc6fc06b550 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc141ce0 con 0x7fc6fc0ab460 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701c62700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc702463700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc6fc06b180 0x7fc6fc1415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44506/0 (socket says 172.21.15.151:44506) 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc702463700 1 -- 172.21.15.151:0/3847791241 learned_addr learned my addr 172.21.15.151:0/3847791241 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701c62700 1 -- 172.21.15.151:0/3847791241 >> v1:172.21.15.172:6789/0 conn(0x7fc6fc06b550 legacy=0x7fc6fc13e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701c62700 1 -- 172.21.15.151:0/3847791241 >> v1:172.21.15.151:6789/0 conn(0x7fc6fc06b180 legacy=0x7fc6fc1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701c62700 1 -- 172.21.15.151:0/3847791241 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0a9b00 con 0x7fc6fc0ab460 2023-09-30T02:00:31.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701c62700 1 --2- 172.21.15.151:0/3847791241 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc0ac390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009e90 tx=0x7fc6f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f182d559c30e0e1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701461700 1 -- 172.21.15.151:0/3847791241 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f800e070 con 0x7fc6fc0ab460 2023-09-30T02:00:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701461700 1 -- 172.21.15.151:0/3847791241 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f800bc30 con 0x7fc6fc0ab460 2023-09-30T02:00:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc701461700 1 -- 172.21.15.151:0/3847791241 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8012430 con 0x7fc6fc0ab460 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 -- 172.21.15.151:0/3847791241 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 msgr2=0x7fc6fc0ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.668+0000 7fc703465700 1 --2- 172.21.15.151:0/3847791241 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc0ac390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009e90 tx=0x7fc6f8009780 comp rx=0 tx=0).stop 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- 172.21.15.151:0/3847791241 shutdown_connections 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 --2- 172.21.15.151:0/3847791241 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc0ac390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- 172.21.15.151:0/3847791241 >> 172.21.15.151:0/3847791241 conn(0x7fc6fc070510 msgr2=0x7fc6fc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- 172.21.15.151:0/3847791241 shutdown_connections 2023-09-30T02:00:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- 172.21.15.151:0/3847791241 wait complete. 2023-09-30T02:00:31.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 Processor -- start 2023-09-30T02:00:31.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- start start 2023-09-30T02:00:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6fc06b550 0x7fc6fc1439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc1390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc143210 con 0x7fc6fc06b180 2023-09-30T02:00:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc1450e0 con 0x7fc6fc06b550 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc702463700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.672+0000 7fc703465700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc143570 con 0x7fc6fc0ab460 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702463700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52146/0 (socket says 172.21.15.151:52146) 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc1390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702463700 1 -- 172.21.15.151:0/752025522 learned_addr learned my addr 172.21.15.151:0/752025522 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc701c62700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6fc06b550 0x7fc6fc1439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 -- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6fc06b550 msgr2=0x7fc6fc1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6fc06b550 0x7fc6fc1439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 -- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 msgr2=0x7fc6fc142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 -- 172.21.15.151:0/752025522 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8009a20 con 0x7fc6fc0ab460 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702463700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc702c64700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc1390f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c4e0 tx=0x7fc6f4007e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400ce50 con 0x7fc6fc0ab460 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4011590 con 0x7fc6fc0ab460 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc0afc10 con 0x7fc6fc0ab460 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4011950 con 0x7fc6fc0ab460 2023-09-30T02:00:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc139850 con 0x7fc6fc0ab460 2023-09-30T02:00:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc0008d0 con 0x7fc6fc0ab460 2023-09-30T02:00:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.676+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc6f4011d00 con 0x7fc6fc0ab460 2023-09-30T02:00:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.680+0000 7fc6f27fc700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 0x7fc6e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.680+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc6f407b820 con 0x7fc6fc0ab460 2023-09-30T02:00:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.680+0000 7fc702463700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 0x7fc6e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.680+0000 7fc702463700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 0x7fc6e8062a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec004900 tx=0x7fc6ec007f70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:31.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.692+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc6f404c000 con 0x7fc6fc0ab460 2023-09-30T02:00:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6fc06d660 con 0x7fc6fc0ab460 2023-09-30T02:00:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc6f27fc700 1 -- 172.21.15.151:0/752025522 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc6f404f620 con 0x7fc6fc0ab460 2023-09-30T02:00:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:31.888 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 msgr2=0x7fc6e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 0x7fc6e8062a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec004900 tx=0x7fc6ec007f70 comp rx=0 tx=0).stop 2023-09-30T02:00:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 msgr2=0x7fc6fc1390f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc1390f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c4e0 tx=0x7fc6f4007e60 comp rx=0 tx=0).stop 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 shutdown_connections 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc6e8060550 0x7fc6e8062a00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc6fc0ab460 0x7fc6fc1390f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6fc06b550 0x7fc6fc1439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 --2- 172.21.15.151:0/752025522 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc6fc06b180 0x7fc6fc142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.884+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 >> 172.21.15.151:0/752025522 conn(0x7fc6fc070510 msgr2=0x7fc6fc13b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.888+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 shutdown_connections 2023-09-30T02:00:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:31.888+0000 7fc703465700 1 -- 172.21.15.151:0/752025522 wait complete. 2023-09-30T02:00:34.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:35.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 Processor -- start 2023-09-30T02:00:35.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 -- start start 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54900b4530 0x7f5490140b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 0x7f5490148140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549006c400 con 0x7f549006e800 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549006c6e0 con 0x7f54900b4530 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54971b9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549006c9c0 con 0x7f54900aec70 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54959b6700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f549006e800 0x7f54900ae560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44514/0 (socket says 172.21.15.151:44514) 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54959b6700 1 -- 172.21.15.151:0/2039126222 learned_addr learned my addr 172.21.15.151:0/2039126222 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:35.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 --2- 172.21.15.151:0/2039126222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 0x7f5490148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54961b7700 1 --2- 172.21.15.151:0/2039126222 >> v2:172.21.15.172:3300/0 conn(0x7f54900b4530 0x7f5490140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54951b5700 1 -- 172.21.15.151:0/2039126222 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011143118 0 0) 0x7f549006c400 con 0x7f549006e800 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54951b5700 1 -- 172.21.15.151:0/2039126222 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c003570 con 0x7f549006e800 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 -- 172.21.15.151:0/2039126222 >> v2:172.21.15.172:3300/0 conn(0x7f54900b4530 msgr2=0x7f5490140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 --2- 172.21.15.151:0/2039126222 >> v2:172.21.15.172:3300/0 conn(0x7f54900b4530 0x7f5490140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 -- 172.21.15.151:0/2039126222 >> v1:172.21.15.151:6789/0 conn(0x7f549006e800 legacy=0x7f54900ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 -- 172.21.15.151:0/2039126222 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549006cca0 con 0x7f54900aec70 2023-09-30T02:00:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54969b8700 1 --2- 172.21.15.151:0/2039126222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 0x7f5490148140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54840098e0 tx=0x7f54840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=908df39e6cb4e723 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:35.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54951b5700 1 -- 172.21.15.151:0/2039126222 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484011630 con 0x7f54900aec70 2023-09-30T02:00:35.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54951b5700 1 -- 172.21.15.151:0/2039126222 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5484011790 con 0x7f54900aec70 2023-09-30T02:00:35.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.016+0000 7f54951b5700 1 -- 172.21.15.151:0/2039126222 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484011b10 con 0x7f54900aec70 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- 172.21.15.151:0/2039126222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 msgr2=0x7f5490148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 --2- 172.21.15.151:0/2039126222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 0x7f5490148140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54840098e0 tx=0x7f54840091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- 172.21.15.151:0/2039126222 shutdown_connections 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 --2- 172.21.15.151:0/2039126222 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54900aec70 0x7f5490148140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 --2- 172.21.15.151:0/2039126222 >> v2:172.21.15.172:3300/0 conn(0x7f54900b4530 0x7f5490140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- 172.21.15.151:0/2039126222 >> 172.21.15.151:0/2039126222 conn(0x7f5490070fe0 msgr2=0x7f54900b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- 172.21.15.151:0/2039126222 shutdown_connections 2023-09-30T02:00:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- 172.21.15.151:0/2039126222 wait complete. 2023-09-30T02:00:35.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 Processor -- start 2023-09-30T02:00:35.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.020+0000 7f54971b9700 1 -- start start 2023-09-30T02:00:35.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54900aec70 0x7f549011c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54900b4530 0x7f549014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549011ce90 con 0x7f54900b4530 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54900aa310 con 0x7f54900aec70 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549011e190 con 0x7f549006e800 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49404/0 (socket says 172.21.15.151:49404) 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 -- 172.21.15.151:0/1463324566 learned_addr learned my addr 172.21.15.151:0/1463324566 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54969b8700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54900b4530 0x7f549014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54959b6700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54900aec70 0x7f549011c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 -- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54900aec70 msgr2=0x7f549011c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54900aec70 0x7f549011c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 -- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54900b4530 msgr2=0x7f549014ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54900b4530 0x7f549014ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 -- 172.21.15.151:0/1463324566 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484009420 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54961b7700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f548800ea00 tx=0x7f5488009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548800c730 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548800c890 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548800cc10 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54900ac110 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549014d410 con 0x7f549006e800 2023-09-30T02:00:35.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.024+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54900008d0 con 0x7f549006e800 2023-09-30T02:00:35.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.028+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5488013470 con 0x7f549006e800 2023-09-30T02:00:35.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.028+0000 7f54827fc700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 0x7f5474062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:35.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.028+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f548807cb80 con 0x7f549006e800 2023-09-30T02:00:35.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.028+0000 7f54959b6700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 0x7f5474062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:35.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.028+0000 7f54959b6700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 0x7f5474062a00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f549014d960 tx=0x7f548c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:35.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.036+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f548804d360 con 0x7f549006e800 2023-09-30T02:00:35.238 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.236+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f549006f690 con 0x7f549006e800 2023-09-30T02:00:35.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.236+0000 7f54827fc700 1 -- 172.21.15.151:0/1463324566 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5488021100 con 0x7f549006e800 2023-09-30T02:00:35.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:35.262 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:35.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 msgr2=0x7f5474062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 0x7f5474062a00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f549014d960 tx=0x7f548c009040 comp rx=0 tx=0).stop 2023-09-30T02:00:35.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 msgr2=0x7f549011bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f548800ea00 tx=0x7f5488009120 comp rx=0 tx=0).stop 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 shutdown_connections 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5474060550 0x7f5474062a00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54900b4530 0x7f549014ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54900aec70 0x7f549011c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 --2- 172.21.15.151:0/1463324566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f549006e800 0x7f549011bd60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 >> 172.21.15.151:0/1463324566 conn(0x7f5490070fe0 msgr2=0x7f549006df50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 shutdown_connections 2023-09-30T02:00:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:35.240+0000 7f54971b9700 1 -- 172.21.15.151:0/1463324566 wait complete. 2023-09-30T02:00:38.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:38.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 Processor -- start 2023-09-30T02:00:38.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 -- start start 2023-09-30T02:00:38.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc840b0960 0x7fcc8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840b0d30 con 0x7fcc8406b3d0 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840ae490 con 0x7fcc840b0960 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc8a261700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8406c690 con 0x7fcc84140c10 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49410/0 (socket says 172.21.15.151:49410) 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 -- 172.21.15.151:0/182567517 learned_addr learned my addr 172.21.15.151:0/182567517 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc837fe700 1 --2- 172.21.15.151:0/182567517 >> v2:172.21.15.172:3300/0 conn(0x7fcc840b0960 0x7fcc8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 -- 172.21.15.151:0/182567517 >> v2:172.21.15.172:3300/0 conn(0x7fcc840b0960 msgr2=0x7fcc8406d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 --2- 172.21.15.151:0/182567517 >> v2:172.21.15.172:3300/0 conn(0x7fcc840b0960 0x7fcc8406d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 -- 172.21.15.151:0/182567517 >> v1:172.21.15.172:6789/0 conn(0x7fcc8406b3d0 legacy=0x7fcc8406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 -- 172.21.15.151:0/182567517 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8406c970 con 0x7fcc84140c10 2023-09-30T02:00:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.372+0000 7fcc83fff700 1 --2- 172.21.15.151:0/182567517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009f20 tx=0x7fcc7c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d20c70fb1592758 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc827fc700 1 -- 172.21.15.151:0/182567517 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c00e070 con 0x7fcc84140c10 2023-09-30T02:00:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc827fc700 1 -- 172.21.15.151:0/182567517 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c00ba80 con 0x7fcc84140c10 2023-09-30T02:00:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc827fc700 1 -- 172.21.15.151:0/182567517 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0092e0 con 0x7fcc84140c10 2023-09-30T02:00:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- 172.21.15.151:0/182567517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 msgr2=0x7fcc84141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 --2- 172.21.15.151:0/182567517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009f20 tx=0x7fcc7c009840 comp rx=0 tx=0).stop 2023-09-30T02:00:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- 172.21.15.151:0/182567517 shutdown_connections 2023-09-30T02:00:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 --2- 172.21.15.151:0/182567517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc84141040 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 --2- 172.21.15.151:0/182567517 >> v2:172.21.15.172:3300/0 conn(0x7fcc840b0960 0x7fcc8406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- 172.21.15.151:0/182567517 >> 172.21.15.151:0/182567517 conn(0x7fcc84070510 msgr2=0x7fcc84070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- 172.21.15.151:0/182567517 shutdown_connections 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- 172.21.15.151:0/182567517 wait complete. 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 Processor -- start 2023-09-30T02:00:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.376+0000 7fcc8a261700 1 -- start start 2023-09-30T02:00:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 0x7fcc840aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 0x7fcc8414d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840ab310 con 0x7fcc8406b3d0 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840acf30 con 0x7fcc840b0960 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840a8850 con 0x7fcc84140c10 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 0x7fcc840aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 0x7fcc840aabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:52160/0 (socket says 172.21.15.151:52160) 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49418/0 (socket says 172.21.15.151:49418) 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc837fe700 1 -- 172.21.15.151:0/4185615248 learned_addr learned my addr 172.21.15.151:0/4185615248 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc82ffd700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 0x7fcc8414d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 -- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 msgr2=0x7fcc8414d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 0x7fcc8414d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 -- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 msgr2=0x7fcc840aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 0x7fcc840aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 -- 172.21.15.151:0/4185615248 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c009ae0 con 0x7fcc84140c10 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc82ffd700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 0x7fcc8414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc83fff700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c015000 tx=0x7fcc7c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0094e0 con 0x7fcc84140c10 2023-09-30T02:00:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c0032f0 con 0x7fcc84140c10 2023-09-30T02:00:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c026460 con 0x7fcc84140c10 2023-09-30T02:00:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8414dfd0 con 0x7fcc84140c10 2023-09-30T02:00:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc840abe80 con 0x7fcc84140c10 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.380+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc840ab0f0 con 0x7fcc84140c10 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.384+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fcc7c003040 con 0x7fcc84140c10 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.384+0000 7fcc6b7fe700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 0x7fcc70062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.384+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcc7c07e390 con 0x7fcc84140c10 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.384+0000 7fcc837fe700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 0x7fcc70062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.384+0000 7fcc837fe700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 0x7fcc70062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc78006fd0 tx=0x7fcc78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.388+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc7c04eb70 con 0x7fcc84140c10 2023-09-30T02:00:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.580+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc840ad450 con 0x7fcc84140c10 2023-09-30T02:00:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc6b7fe700 1 -- 172.21.15.151:0/4185615248 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc840ad450 con 0x7fcc84140c10 2023-09-30T02:00:38.587 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:38.587 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:38.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 msgr2=0x7fcc70062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 0x7fcc70062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc78006fd0 tx=0x7fcc78008040 comp rx=0 tx=0).stop 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 msgr2=0x7fcc840ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c015000 tx=0x7fcc7c018000 comp rx=0 tx=0).stop 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 shutdown_connections 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fcc70060550 0x7fcc70062a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcc84140c10 0x7fcc840ab7a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc840b0960 0x7fcc8414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 --2- 172.21.15.151:0/4185615248 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcc8406b3d0 0x7fcc840aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:38.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 >> 172.21.15.151:0/4185615248 conn(0x7fcc84070510 msgr2=0x7fcc8413ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:38.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 shutdown_connections 2023-09-30T02:00:38.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:38.584+0000 7fcc8a261700 1 -- 172.21.15.151:0/4185615248 wait complete. 2023-09-30T02:00:41.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:41.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.708+0000 7fc7adcf2700 1 Processor -- start 2023-09-30T02:00:41.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.708+0000 7fc7adcf2700 1 -- start start 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7adcf2700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 0x7fc7a80ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7adcf2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a806ecb0 con 0x7fc7a80ac230 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7adcf2700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a8140c20 con 0x7fc7a8141420 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7adcf2700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a8140f00 con 0x7fc7a806e350 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc7a80ac230 0x7fc7a813d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44616/0 (socket says 172.21.15.151:44616) 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 -- 172.21.15.151:0/538842714 learned_addr learned my addr 172.21.15.151:0/538842714 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 0x7fc7a80ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 msgr2=0x7fc7a80ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 msgr2=0x7fc7a80ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:00:41.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 0x7fc7a80ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 0x7fc7a80ac600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526669973 0 0) 0x7fc7a806ecb0 con 0x7fc7a80ac230 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003570 con 0x7fc7a80ac230 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454141833 0 0) 0x7fc7a8140c20 con 0x7fc7a8141420 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a806ecb0 con 0x7fc7a8141420 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584530336 0 0) 0x7fc794003570 con 0x7fc7a80ac230 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 msgr2=0x7fc7a80ac600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 --2- 172.21.15.151:0/538842714 >> v2:172.21.15.172:3300/0 conn(0x7fc7a806e350 0x7fc7a80ac600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 >> v1:172.21.15.172:6789/0 conn(0x7fc7a8141420 legacy=0x7fc7a806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80ab2c0 con 0x7fc7a80ac230 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7fc790001470 con 0x7fc7a80ac230 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 >> v1:172.21.15.151:6789/0 conn(0x7fc7a80ac230 legacy=0x7fc7a813d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 0x7fc794003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc794004810 0x7fc794004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794005230 0x7fc79400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8140f00 con 0x7fc794003bf0 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79400c160 con 0x7fc794004810 2023-09-30T02:00:41.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79400c320 con 0x7fc794005230 2023-09-30T02:00:41.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 0x7fc794003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794005230 0x7fc79400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a7fff700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc794004810 0x7fc794004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 -- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794005230 msgr2=0x7fc79400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:41.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794005230 0x7fc79400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 -- 172.21.15.151:0/538842714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc794004810 msgr2=0x7fc794004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc794004810 0x7fc794004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/538842714 reap_dead start 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 -- 172.21.15.151:0/538842714 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794003570 con 0x7fc794003bf0 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7accf0700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 0x7fc794003fe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c2d0 tx=0x7fc790001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7820b0bc73e54f74 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000cbc0 con 0x7fc794003bf0 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79000cd20 con 0x7fc794003bf0 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.712+0000 7fc7a77fe700 1 -- 172.21.15.151:0/538842714 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790005b70 con 0x7fc794003bf0 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 msgr2=0x7fc794003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 0x7fc794003fe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c2d0 tx=0x7fc790001e10 comp rx=0 tx=0).stop 2023-09-30T02:00:41.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- 172.21.15.151:0/538842714 shutdown_connections 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/538842714 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc794003bf0 0x7fc794003fe0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- 172.21.15.151:0/538842714 >> 172.21.15.151:0/538842714 conn(0x7fc7a8070510 msgr2=0x7fc7a8070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- 172.21.15.151:0/538842714 shutdown_connections 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- 172.21.15.151:0/538842714 wait complete. 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 Processor -- start 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- start start 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794003bf0 0x7fc7a8117590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 0x7fc7a8150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8150900 con 0x7fc7a81185f0 2023-09-30T02:00:41.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8139d10 con 0x7fc7a81181e0 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7adcf2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8139e80 con 0x7fc794003bf0 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36384/0 (socket says 172.21.15.151:36384) 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/28358598 learned_addr learned my addr 172.21.15.151:0/28358598 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7accf0700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794003bf0 0x7fc7a8117590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7a7fff700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 0x7fc7a8150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794003bf0 msgr2=0x7fc7a8117590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794003bf0 0x7fc7a8117590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 msgr2=0x7fc7a8150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 0x7fc7a8150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7ad4f1700 1 -- 172.21.15.151:0/28358598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8137f60 con 0x7fc7a81185f0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.716+0000 7fc7a7fff700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 0x7fc7a8150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7ad4f1700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc7a806e790 tx=0x7fc7a0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0006a90 con 0x7fc7a81185f0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a0016690 con 0x7fc7a81185f0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a81512d0 con 0x7fc7a81185f0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0016a50 con 0x7fc7a81185f0 2023-09-30T02:00:41.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a814d7a0 con 0x7fc7a81185f0 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8150ab0 con 0x7fc7a81185f0 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc7a0014430 con 0x7fc7a81185f0 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 0x7fc788062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc7a007c9f0 con 0x7fc7a81185f0 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.720+0000 7fc7accf0700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 0x7fc788062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.724+0000 7fc7accf0700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 0x7fc788062a30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc790008fd0 tx=0x7fc790015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:41.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.732+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc7a8150ab0 con 0x7fc7a81185f0 2023-09-30T02:00:41.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7a8150ab0 con 0x7fc7a81185f0 2023-09-30T02:00:41.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7a4ff9700 1 -- 172.21.15.151:0/28358598 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc7a8150ab0 con 0x7fc7a81185f0 2023-09-30T02:00:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:41.932 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 msgr2=0x7fc788062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 0x7fc788062a30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc790008fd0 tx=0x7fc790015000 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 msgr2=0x7fc7a814d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc7a806e790 tx=0x7fc7a0008040 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.928+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 shutdown_connections 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fc788060580 0x7fc788062a30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81185f0 0x7fc7a814d260 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7a81181e0 0x7fc7a8150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 --2- 172.21.15.151:0/28358598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc794003bf0 0x7fc7a8117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 >> 172.21.15.151:0/28358598 conn(0x7fc7a8070510 msgr2=0x7fc7a813d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:41.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 shutdown_connections 2023-09-30T02:00:41.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:41.932+0000 7fc7adcf2700 1 -- 172.21.15.151:0/28358598 wait complete. 2023-09-30T02:00:44.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:45.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 Processor -- start 2023-09-30T02:00:45.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 -- start start 2023-09-30T02:00:45.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3c40675a0 0x7fd3c40a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd3c413f780 0x7fd3c413fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c40aa370 con 0x7fd3c4140b10 2023-09-30T02:00:45.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c4067970 con 0x7fd3c413f780 2023-09-30T02:00:45.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca89c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c4141680 con 0x7fd3c40675a0 2023-09-30T02:00:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd3c413f780 0x7fd3c413fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c9099700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd3c4140b10 0x7fd3c4140f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44630/0 (socket says 172.21.15.151:44630) 2023-09-30T02:00:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c9099700 1 -- 172.21.15.151:0/537126552 learned_addr learned my addr 172.21.15.151:0/537126552 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c989a700 1 --2- 172.21.15.151:0/537126552 >> v2:172.21.15.172:3300/0 conn(0x7fd3c40675a0 0x7fd3c40a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 -- 172.21.15.151:0/537126552 >> v2:172.21.15.172:3300/0 conn(0x7fd3c40675a0 msgr2=0x7fd3c40a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 --2- 172.21.15.151:0/537126552 >> v2:172.21.15.172:3300/0 conn(0x7fd3c40675a0 0x7fd3c40a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 -- 172.21.15.151:0/537126552 >> v1:172.21.15.151:6789/0 conn(0x7fd3c4140b10 legacy=0x7fd3c4140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 -- 172.21.15.151:0/537126552 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c40aa0b0 con 0x7fd3c413f780 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c8898700 1 -- 172.21.15.151:0/537126552 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210099355 0 0) 0x7fd3c40aa370 con 0x7fd3c4140b10 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3ca09b700 1 --2- 172.21.15.151:0/537126552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c413f780 0x7fd3c413fbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40ab0d0 tx=0x7fd3bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ba640ac113427d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c8898700 1 -- 172.21.15.151:0/537126552 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00e070 con 0x7fd3c413f780 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.060+0000 7fd3c8898700 1 -- 172.21.15.151:0/537126552 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc00bbf0 con 0x7fd3c413f780 2023-09-30T02:00:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3c8898700 1 -- 172.21.15.151:0/537126552 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc012430 con 0x7fd3c413f780 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- 172.21.15.151:0/537126552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c413f780 msgr2=0x7fd3c413fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/537126552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c413f780 0x7fd3c413fbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40ab0d0 tx=0x7fd3bc009780 comp rx=0 tx=0).stop 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- 172.21.15.151:0/537126552 shutdown_connections 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/537126552 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c413f780 0x7fd3c413fbe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/537126552 >> v2:172.21.15.172:3300/0 conn(0x7fd3c40675a0 0x7fd3c40a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- 172.21.15.151:0/537126552 >> 172.21.15.151:0/537126552 conn(0x7fd3c40b08c0 msgr2=0x7fd3c40b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- 172.21.15.151:0/537126552 shutdown_connections 2023-09-30T02:00:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- 172.21.15.151:0/537126552 wait complete. 2023-09-30T02:00:45.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 Processor -- start 2023-09-30T02:00:45.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.064+0000 7fd3ca89c700 1 -- start start 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 0x7fd3c406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 0x7fd3c41418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c406d650 con 0x7fd3c40675a0 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c41435f0 con 0x7fd3c4140b10 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c413a0b0 con 0x7fd3c413f780 2023-09-30T02:00:45.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c989a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 0x7fd3c406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:41198/0 (socket says 172.21.15.151:41198) 2023-09-30T02:00:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 -- 172.21.15.151:0/1728708798 learned_addr learned my addr 172.21.15.151:0/1728708798 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca09b700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 0x7fd3c41418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 -- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 msgr2=0x7fd3c41418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 0x7fd3c41418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 -- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 msgr2=0x7fd3c406cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 0x7fd3c406cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 -- 172.21.15.151:0/1728708798 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc009a20 con 0x7fd3c413f780 2023-09-30T02:00:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c989a700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 0x7fd3c406cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca09b700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 0x7fd3c41418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3c9099700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c5b0 tx=0x7fd3b8001070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8005b30 con 0x7fd3c413f780 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b8005c90 con 0x7fd3c413f780 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c406dae0 con 0x7fd3c413f780 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8007300 con 0x7fd3c413f780 2023-09-30T02:00:45.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c4142050 con 0x7fd3c413f780 2023-09-30T02:00:45.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.068+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c406d7c0 con 0x7fd3c413f780 2023-09-30T02:00:45.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.072+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd3b801b3b0 con 0x7fd3c413f780 2023-09-30T02:00:45.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.072+0000 7fd3b5ffb700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 0x7fd3a8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:45.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.072+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd3b807bb00 con 0x7fd3c413f780 2023-09-30T02:00:45.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.072+0000 7fd3c989a700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 0x7fd3a8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:45.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.072+0000 7fd3c989a700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 0x7fd3a8062a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011480 tx=0x7fd3c00091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:45.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.080+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd3b804c2e0 con 0x7fd3c413f780 2023-09-30T02:00:45.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.272+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd3c40008d0 con 0x7fd3c413f780 2023-09-30T02:00:45.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3b5ffb700 1 -- 172.21.15.151:0/1728708798 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd3b804f900 con 0x7fd3c413f780 2023-09-30T02:00:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:45.279 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:45.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 msgr2=0x7fd3a8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 0x7fd3a8062a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011480 tx=0x7fd3c00091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 msgr2=0x7fd3c406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c5b0 tx=0x7fd3b8001070 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 shutdown_connections 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd3a8060550 0x7fd3a8062a00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3c4140b10 0x7fd3c41418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3c413f780 0x7fd3c406c920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 --2- 172.21.15.151:0/1728708798 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd3c40675a0 0x7fd3c406cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 >> 172.21.15.151:0/1728708798 conn(0x7fd3c40b08c0 msgr2=0x7fd3c40a9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:45.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 shutdown_connections 2023-09-30T02:00:45.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:45.276+0000 7fd3ca89c700 1 -- 172.21.15.151:0/1728708798 wait complete. 2023-09-30T02:00:48.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:48.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.404+0000 7f2f9d089700 1 Processor -- start 2023-09-30T02:00:48.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.404+0000 7f2f9d089700 1 -- start start 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2f980706b0 0x7f2f9813d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f9806e910 0x7f2f980ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 0x7f2f9813c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f980a5ff0 con 0x7f2f9806e910 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f98070a80 con 0x7f2f980706b0 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980ac900 con 0x7f2f980a48f0 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f9806e910 0x7f2f980ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 0x7f2f9813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f9806e910 0x7f2f980ac3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36426/0 (socket says 172.21.15.151:36426) 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 -- 172.21.15.151:0/4201239382 learned_addr learned my addr 172.21.15.151:0/4201239382 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f96d9d700 1 --2- 172.21.15.151:0/4201239382 >> v2:172.21.15.172:3300/0 conn(0x7f2f980706b0 0x7f2f9813d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 -- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 msgr2=0x7f2f9813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 0x7f2f9813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 -- 172.21.15.151:0/4201239382 >> v2:172.21.15.172:3300/0 conn(0x7f2f980706b0 msgr2=0x7f2f9813d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/4201239382 >> v2:172.21.15.172:3300/0 conn(0x7f2f980706b0 0x7f2f9813d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 -- 172.21.15.151:0/4201239382 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980aca70 con 0x7f2f9806e910 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9759e700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 0x7f2f9813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f980ac3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f98070d20 tx=0x7f2f80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8399dea7c7a1391 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9659c700 1 -- 172.21.15.151:0/4201239382 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000b640 con 0x7f2f9806e910 2023-09-30T02:00:48.413 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9659c700 1 -- 172.21.15.151:0/4201239382 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8000b7a0 con 0x7f2f9806e910 2023-09-30T02:00:48.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9659c700 1 -- 172.21.15.151:0/4201239382 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000bb60 con 0x7f2f9806e910 2023-09-30T02:00:48.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 msgr2=0x7f2f980ac3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f980ac3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f98070d20 tx=0x7f2f80009250 comp rx=0 tx=0).stop 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- 172.21.15.151:0/4201239382 shutdown_connections 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980a48f0 0x7f2f9813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- 172.21.15.151:0/4201239382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f980ac3c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 --2- 172.21.15.151:0/4201239382 >> v2:172.21.15.172:3300/0 conn(0x7f2f980706b0 0x7f2f9813d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- 172.21.15.151:0/4201239382 >> 172.21.15.151:0/4201239382 conn(0x7f2f98067510 msgr2=0x7f2f98067910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- 172.21.15.151:0/4201239382 shutdown_connections 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.408+0000 7f2f9d089700 1 -- 172.21.15.151:0/4201239382 wait complete. 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 Processor -- start 2023-09-30T02:00:48.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- start start 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 0x7f2f981506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 0x7f2f980a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f98150fd0 con 0x7f2f9806e910 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9813f0c0 con 0x7f2f980a48f0 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9813f3a0 con 0x7f2f980706b0 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f8ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 0x7f2f981506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36434/0 (socket says 172.21.15.151:36434) 2023-09-30T02:00:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 -- 172.21.15.151:0/3265232674 learned_addr learned my addr 172.21.15.151:0/3265232674 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9759e700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 0x7f2f980a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 -- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 msgr2=0x7f2f981506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:48.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 0x7f2f981506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 -- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 msgr2=0x7f2f980a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 0x7f2f980a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 -- 172.21.15.151:0/3265232674 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f90009a20 con 0x7f2f9806e910 2023-09-30T02:00:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9759e700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 0x7f2f980a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 0x7f2f981506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f96d9d700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800ec80 tx=0x7f2f880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800c850 con 0x7f2f9806e910 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8800c9b0 con 0x7f2f9806e910 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f800094f0 con 0x7f2f9806e910 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800cd50 con 0x7f2f9806e910 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f980a9a80 con 0x7f2f9806e910 2023-09-30T02:00:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.412+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98150d50 con 0x7f2f9806e910 2023-09-30T02:00:48.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.416+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2f88013680 con 0x7f2f9806e910 2023-09-30T02:00:48.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.416+0000 7f2f8effd700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 0x7f2f78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:48.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.416+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2f8807c860 con 0x7f2f9806e910 2023-09-30T02:00:48.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.416+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 0x7f2f78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:48.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.416+0000 7f2f8ffff700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 0x7f2f78062a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f98070d20 tx=0x7f2f80017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:48.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.420+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f8804d040 con 0x7f2f9806e910 2023-09-30T02:00:48.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.620+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f980670a0 con 0x7f2f9806e910 2023-09-30T02:00:48.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.620+0000 7f2f8effd700 1 -- 172.21.15.151:0/3265232674 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2f88050660 con 0x7f2f9806e910 2023-09-30T02:00:48.625 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:48.626 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 msgr2=0x7f2f78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 0x7f2f78062a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f98070d20 tx=0x7f2f80017000 comp rx=0 tx=0).stop 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 msgr2=0x7f2f9806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800ec80 tx=0x7f2f880091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 shutdown_connections 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2f78060550 0x7f2f78062a00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f980a48f0 0x7f2f980a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f980706b0 0x7f2f981506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 --2- 172.21.15.151:0/3265232674 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f9806e910 0x7f2f9806dcb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:48.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 >> 172.21.15.151:0/3265232674 conn(0x7f2f98067510 msgr2=0x7f2f980aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:48.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 shutdown_connections 2023-09-30T02:00:48.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:48.624+0000 7f2f9d089700 1 -- 172.21.15.151:0/3265232674 wait complete. 2023-09-30T02:00:51.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.748+0000 7f851b1c8700 1 Processor -- start 2023-09-30T02:00:51.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.748+0000 7f851b1c8700 1 -- start start 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851b1c8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85140ac140 0x7f85140aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851b1c8700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851406ecd0 con 0x7f85140ac510 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851b1c8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8514138c00 con 0x7f851406e870 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851b1c8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85140abbd0 con 0x7f85140ac140 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85199c5700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f85140ac510 0x7f851413afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35366/0 (socket says 172.21.15.151:35366) 2023-09-30T02:00:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85199c5700 1 -- 172.21.15.151:0/1146133773 learned_addr learned my addr 172.21.15.151:0/1146133773 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 --2- 172.21.15.151:0/1146133773 >> v2:172.21.15.172:3300/0 conn(0x7f85140ac140 0x7f85140aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422330223 0 0) 0x7f851406ecd0 con 0x7f85140ac510 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc003570 con 0x7f85140ac510 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460932110 0 0) 0x7f8514138c00 con 0x7f851406e870 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851406ecd0 con 0x7f851406e870 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664243998 0 0) 0x7f84fc003570 con 0x7f85140ac510 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 >> v2:172.21.15.172:3300/0 conn(0x7f85140ac140 msgr2=0x7f85140aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 --2- 172.21.15.151:0/1146133773 >> v2:172.21.15.172:3300/0 conn(0x7f85140ac140 0x7f85140aa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 >> v1:172.21.15.172:6789/0 conn(0x7f851406e870 legacy=0x7f85140a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140abdd0 con 0x7f85140ac510 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f85040017f0 con 0x7f85140ac510 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 >> v1:172.21.15.151:6789/0 conn(0x7f85140ac510 legacy=0x7f851413afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 0x7f84fc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc004810 0x7f84fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84fc0051c0 0x7f84fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140abbd0 con 0x7f84fc003bf0 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc00c0b0 con 0x7f84fc0051c0 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc00c270 con 0x7f84fc004810 2023-09-30T02:00:51.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 0x7f84fc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85199c5700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc004810 0x7f84fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a9c7700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84fc0051c0 0x7f84fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 -- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc004810 msgr2=0x7f84fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc004810 0x7f84fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 -- 172.21.15.151:0/1146133773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84fc0051c0 msgr2=0x7f84fc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f84fc0051c0 0x7f84fc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 -- 172.21.15.151:0/1146133773 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc003570 con 0x7f84fc003bf0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a9c7700 1 -- 172.21.15.151:0/1146133773 reap_dead start 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f851a1c6700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 0x7f84fc003fe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8508009c00 tx=0x7f85080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5bf060aa968253 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800e710 con 0x7f84fc003bf0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850800e870 con 0x7f84fc003bf0 2023-09-30T02:00:51.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.752+0000 7f85191c4700 1 -- 172.21.15.151:0/1146133773 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800ec30 con 0x7f84fc003bf0 2023-09-30T02:00:51.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 msgr2=0x7f84fc003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 0x7f84fc003fe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8508009c00 tx=0x7f85080091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- 172.21.15.151:0/1146133773 shutdown_connections 2023-09-30T02:00:51.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 --2- 172.21.15.151:0/1146133773 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84fc003bf0 0x7f84fc003fe0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- 172.21.15.151:0/1146133773 >> 172.21.15.151:0/1146133773 conn(0x7f8514070510 msgr2=0x7f8514070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:51.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- 172.21.15.151:0/1146133773 shutdown_connections 2023-09-30T02:00:51.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- 172.21.15.151:0/1146133773 wait complete. 2023-09-30T02:00:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 Processor -- start 2023-09-30T02:00:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- start start 2023-09-30T02:00:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 0x7f851414b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85140a7d90 0x7f851414be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514149200 con 0x7f85140a7940 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514148c50 con 0x7f85140a7d90 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851b1c8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8514148880 con 0x7f84fc003bf0 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851a1c6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f85199c5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 0x7f851414b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851a1c6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38688/0 (socket says 172.21.15.151:38688) 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851a1c6700 1 -- 172.21.15.151:0/2679629173 learned_addr learned my addr 172.21.15.151:0/2679629173 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.756+0000 7f851a9c7700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85140a7d90 0x7f851414be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 -- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 msgr2=0x7f8514148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 -- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85140a7d90 msgr2=0x7f851414be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85140a7d90 0x7f851414be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 -- 172.21.15.151:0/2679629173 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140a86f0 con 0x7f85140a7940 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f851a1c6700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85199c5700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 0x7f851414b8d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f851405ec40 tx=0x7f8504002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400aa80 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850400ac00 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140a88e0 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040133f0 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851414c570 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851414c700 con 0x7f85140a7940 2023-09-30T02:00:51.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f85040137e0 con 0x7f85140a7940 2023-09-30T02:00:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 0x7f8500062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f850407d0e0 con 0x7f85140a7940 2023-09-30T02:00:51.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.760+0000 7f851a1c6700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 0x7f8500062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.764+0000 7f851a1c6700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 0x7f8500062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8508006ed0 tx=0x7f850801a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:51.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.768+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f850404d8c0 con 0x7f85140a7940 2023-09-30T02:00:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.968+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f85140008d0 con 0x7f85140a7940 2023-09-30T02:00:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.968+0000 7f85127fc700 1 -- 172.21.15.151:0/2679629173 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8504017080 con 0x7f85140a7940 2023-09-30T02:00:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:51.974 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:51.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 msgr2=0x7f8500062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 0x7f8500062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8508006ed0 tx=0x7f850801a000 comp rx=0 tx=0).stop 2023-09-30T02:00:51.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 msgr2=0x7f851414b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:51.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 0x7f851414b8d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f851405ec40 tx=0x7f8504002190 comp rx=0 tx=0).stop 2023-09-30T02:00:51.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 shutdown_connections 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8500060550 0x7f8500062a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85140a7d90 0x7f851414be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85140a7940 0x7f851414b8d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 --2- 172.21.15.151:0/2679629173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84fc003bf0 0x7f8514148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 >> 172.21.15.151:0/2679629173 conn(0x7f8514070510 msgr2=0x7f8514137b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 shutdown_connections 2023-09-30T02:00:51.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:51.972+0000 7f851b1c8700 1 -- 172.21.15.151:0/2679629173 wait complete. 2023-09-30T02:00:54.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:55.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.092+0000 7f24f6d10700 1 Processor -- start 2023-09-30T02:00:55.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- start start 2023-09-30T02:00:55.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:55.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f006c620 con 0x7f24f00ad600 2023-09-30T02:00:55.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f006c900 con 0x7f24f00a7680 2023-09-30T02:00:55.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f006d610 con 0x7f24f006de10 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38700/0 (socket says 172.21.15.151:38700) 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 -- 172.21.15.151:0/2714735561 learned_addr learned my addr 172.21.15.151:0/2714735561 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f4aac700 1 --1- 172.21.15.151:0/2714735561 >> v1:172.21.15.151:6789/0 conn(0x7f24f00ad600 0x7f24f00ab900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35368/0 (socket says 172.21.15.151:35368) 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 -- 172.21.15.151:0/2714735561 >> v1:172.21.15.172:6789/0 conn(0x7f24f00a7680 legacy=0x7f24f0140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 -- 172.21.15.151:0/2714735561 >> v1:172.21.15.151:6789/0 conn(0x7f24f00ad600 legacy=0x7f24f00ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 -- 172.21.15.151:0/2714735561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f006d8f0 con 0x7f24f006de10 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f52ad700 1 --2- 172.21.15.151:0/2714735561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f24e00099e0 tx=0x7f24e0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102b3e7c4d7a17ba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:55.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24ef7fe700 1 -- 172.21.15.151:0/2714735561 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000b820 con 0x7f24f006de10 2023-09-30T02:00:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24ef7fe700 1 -- 172.21.15.151:0/2714735561 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e000b980 con 0x7f24f006de10 2023-09-30T02:00:55.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24ef7fe700 1 -- 172.21.15.151:0/2714735561 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000bd20 con 0x7f24f006de10 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- 172.21.15.151:0/2714735561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 msgr2=0x7f24f0141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 --2- 172.21.15.151:0/2714735561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f24e00099e0 tx=0x7f24e0009250 comp rx=0 tx=0).stop 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- 172.21.15.151:0/2714735561 shutdown_connections 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 --2- 172.21.15.151:0/2714735561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f0141280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- 172.21.15.151:0/2714735561 >> 172.21.15.151:0/2714735561 conn(0x7f24f0070510 msgr2=0x7f24f00708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- 172.21.15.151:0/2714735561 shutdown_connections 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.096+0000 7f24f6d10700 1 -- 172.21.15.151:0/2714735561 wait complete. 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 Processor -- start 2023-09-30T02:00:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- start start 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f00af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24f00a7680 0x7f24f014c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f00b0050 con 0x7f24f00ad600 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f00afd00 con 0x7f24f00a7680 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f0144510 con 0x7f24f006de10 2023-09-30T02:00:55.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f4aac700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f00af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45308/0 (socket says 172.21.15.151:45308) 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 -- 172.21.15.151:0/3576085924 learned_addr learned my addr 172.21.15.151:0/3576085924 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24effff700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24f00a7680 0x7f24f014c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 -- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 msgr2=0x7f24f00af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f00af7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 -- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24f00a7680 msgr2=0x7f24f014c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24f00a7680 0x7f24f014c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 -- 172.21.15.151:0/3576085924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00094f0 con 0x7f24f00ad600 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f52ad700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009290 tx=0x7f24e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:55.106 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0011e60 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0020070 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f00ae860 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e001b6a0 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.100+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f00aee00 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f00b0200 con 0x7f24f00ad600 2023-09-30T02:00:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f24e0022030 con 0x7f24f00ad600 2023-09-30T02:00:55.108 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24ecff9700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 0x7f24d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:55.109 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f24e007e6c0 con 0x7f24f00ad600 2023-09-30T02:00:55.109 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24f4aac700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 0x7f24d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:55.109 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.104+0000 7f24f4aac700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 0x7f24d0062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f24e8002020 tx=0x7f24e8001a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:55.116 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.112+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f24e004ee00 con 0x7f24f00ad600 2023-09-30T02:00:55.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.312+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24f00b0200 con 0x7f24f00ad600 2023-09-30T02:00:55.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.316+0000 7f24ecff9700 1 -- 172.21.15.151:0/3576085924 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f24e0052420 con 0x7f24f00ad600 2023-09-30T02:00:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:55.321 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 msgr2=0x7f24d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 0x7f24d0062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f24e8002020 tx=0x7f24e8001a30 comp rx=0 tx=0).stop 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 msgr2=0x7f24f014cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009290 tx=0x7f24e0016000 comp rx=0 tx=0).stop 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 shutdown_connections 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f24d0060550 0x7f24d0062a00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24f00ad600 0x7f24f014cd70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24f00a7680 0x7f24f014c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 --2- 172.21.15.151:0/3576085924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24f006de10 0x7f24f00af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 >> 172.21.15.151:0/3576085924 conn(0x7f24f0070510 msgr2=0x7f24f013efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 shutdown_connections 2023-09-30T02:00:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:55.320+0000 7f24f6d10700 1 -- 172.21.15.151:0/3576085924 wait complete. 2023-09-30T02:00:58.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:00:58.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 Processor -- start 2023-09-30T02:00:58.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- start start 2023-09-30T02:00:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4d20136600 0x7f4d20136a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2006b820 con 0x7f4d2006c500 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2006d5e0 con 0x7f4d20067a70 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d20067e40 con 0x7f4d20136600 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d258d2700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4d2006c500 0x7f4d2006bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35370/0 (socket says 172.21.15.151:35370) 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d258d2700 1 -- 172.21.15.151:0/611350925 learned_addr learned my addr 172.21.15.151:0/611350925 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d268d4700 1 --2- 172.21.15.151:0/611350925 >> v2:172.21.15.151:3300/0 conn(0x7f4d20136600 0x7f4d20136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269170626 0 0) 0x7f4d2006b820 con 0x7f4d2006c500 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c003570 con 0x7f4d2006c500 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752719343 0 0) 0x7f4d2006d5e0 con 0x7f4d20067a70 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2006b820 con 0x7f4d20067a70 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d268d4700 1 -- 172.21.15.151:0/611350925 >> v1:172.21.15.172:6789/0 conn(0x7f4d20067a70 legacy=0x7f4d2006df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d268d4700 1 -- 172.21.15.151:0/611350925 >> v1:172.21.15.151:6789/0 conn(0x7f4d2006c500 legacy=0x7f4d2006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d268d4700 1 -- 172.21.15.151:0/611350925 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2006db30 con 0x7f4d20136600 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d268d4700 1 --2- 172.21.15.151:0/611350925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20136600 0x7f4d20136a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400aa90 tx=0x7f4d1400a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9234e99456d4e3f1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400c970 con 0x7f4d20136600 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1400cad0 con 0x7f4d20136600 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d250d1700 1 -- 172.21.15.151:0/611350925 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400ce50 con 0x7f4d20136600 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- 172.21.15.151:0/611350925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20136600 msgr2=0x7f4d20136a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 --2- 172.21.15.151:0/611350925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20136600 0x7f4d20136a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400aa90 tx=0x7f4d1400a330 comp rx=0 tx=0).stop 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 -- 172.21.15.151:0/611350925 shutdown_connections 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.448+0000 7f4d270d5700 1 --2- 172.21.15.151:0/611350925 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20136600 0x7f4d20136a80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- 172.21.15.151:0/611350925 >> 172.21.15.151:0/611350925 conn(0x7f4d200b03e0 msgr2=0x7f4d200b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- 172.21.15.151:0/611350925 shutdown_connections 2023-09-30T02:00:58.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- 172.21.15.151:0/611350925 wait complete. 2023-09-30T02:00:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 Processor -- start 2023-09-30T02:00:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- start start 2023-09-30T02:00:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d2006c500 0x7f4d2013bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200aa280 con 0x7f4d20067a70 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200aa560 con 0x7f4d2006c500 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200a8d40 con 0x7f4d20136600 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d260d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d260d3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45348/0 (socket says 172.21.15.151:45348) 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38732/0 (socket says 172.21.15.151:38732) 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d260d3700 1 -- 172.21.15.151:0/721747326 learned_addr learned my addr 172.21.15.151:0/721747326 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d258d2700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d2006c500 0x7f4d2013bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 -- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d2006c500 msgr2=0x7f4d2013bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d2006c500 0x7f4d2013bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 -- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 msgr2=0x7f4d200af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 -- 172.21.15.151:0/721747326 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1400a5d0 con 0x7f4d20136600 2023-09-30T02:00:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d260d3700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d268d4700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d14017000 tx=0x7f4d1401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1401c040 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d14010600 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d200a9020 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400c970 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.452+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d200a9760 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200008d0 con 0x7f4d20136600 2023-09-30T02:00:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4d14010c50 con 0x7f4d20136600 2023-09-30T02:00:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d127fc700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 0x7f4d04062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4d1407d410 con 0x7f4d20136600 2023-09-30T02:00:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d260d3700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 0x7f4d04062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:58.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.456+0000 7f4d260d3700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 0x7f4d04062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d18007fd0 tx=0x7f4d18002820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:58.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.464+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4d1404dbf0 con 0x7f4d20136600 2023-09-30T02:00:58.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.656+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d200ac820 con 0x7f4d20136600 2023-09-30T02:00:58.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.656+0000 7f4d127fc700 1 -- 172.21.15.151:0/721747326 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4d14051210 con 0x7f4d20136600 2023-09-30T02:00:58.662 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:00:58.662 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:00:58.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 msgr2=0x7f4d04062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 0x7f4d04062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d18007fd0 tx=0x7f4d18002820 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 msgr2=0x7f4d2013c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d14017000 tx=0x7f4d1401a000 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 shutdown_connections 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4d04060550 0x7f4d04062a00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d20136600 0x7f4d2013c0c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4d2006c500 0x7f4d2013bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 --2- 172.21.15.151:0/721747326 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d20067a70 0x7f4d200af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 >> 172.21.15.151:0/721747326 conn(0x7f4d200b03e0 msgr2=0x7f4d20070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 shutdown_connections 2023-09-30T02:00:58.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:00:58.660+0000 7f4d270d5700 1 -- 172.21.15.151:0/721747326 wait complete. 2023-09-30T02:01:01.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:01.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 Processor -- start 2023-09-30T02:01:01.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 -- start start 2023-09-30T02:01:01.789 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdc5c06e660 0x7fdc5c13d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c067e80 con 0x7fdc5c067ab0 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c06eb80 con 0x7fdc5c1364e0 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc61d10700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c06da30 con 0x7fdc5c06e660 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc60d0e700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fdc5c067ab0 0x7fdc5c13cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41352/0 (socket says 172.21.15.151:41352) 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc60d0e700 1 -- 172.21.15.151:0/1045126339 learned_addr learned my addr 172.21.15.151:0/1045126339 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:01.790 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc5bfff700 1 --2- 172.21.15.151:0/1045126339 >> v2:172.21.15.172:3300/0 conn(0x7fdc5c06e660 0x7fdc5c13d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:01.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.784+0000 7fdc5b7fe700 1 -- 172.21.15.151:0/1045126339 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455521614 0 0) 0x7fdc5c067e80 con 0x7fdc5c067ab0 2023-09-30T02:01:01.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5b7fe700 1 -- 172.21.15.151:0/1045126339 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48004680 con 0x7fdc5c067ab0 2023-09-30T02:01:01.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5bfff700 1 -- 172.21.15.151:0/1045126339 >> v1:172.21.15.172:6789/0 conn(0x7fdc5c1364e0 legacy=0x7fdc5c136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5bfff700 1 -- 172.21.15.151:0/1045126339 >> v1:172.21.15.151:6789/0 conn(0x7fdc5c067ab0 legacy=0x7fdc5c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5bfff700 1 -- 172.21.15.151:0/1045126339 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c13d9e0 con 0x7fdc5c06e660 2023-09-30T02:01:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5bfff700 1 --2- 172.21.15.151:0/1045126339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c06e660 0x7fdc5c13d4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48bba2528a4e748d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5b7fe700 1 -- 172.21.15.151:0/1045126339 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00b640 con 0x7fdc5c06e660 2023-09-30T02:01:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5b7fe700 1 -- 172.21.15.151:0/1045126339 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c00b7a0 con 0x7fdc5c06e660 2023-09-30T02:01:01.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc5b7fe700 1 -- 172.21.15.151:0/1045126339 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00bb20 con 0x7fdc5c06e660 2023-09-30T02:01:01.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 -- 172.21.15.151:0/1045126339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c06e660 msgr2=0x7fdc5c13d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 --2- 172.21.15.151:0/1045126339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c06e660 0x7fdc5c13d4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 -- 172.21.15.151:0/1045126339 shutdown_connections 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 --2- 172.21.15.151:0/1045126339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c06e660 0x7fdc5c13d4a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 -- 172.21.15.151:0/1045126339 >> 172.21.15.151:0/1045126339 conn(0x7fdc5c0b03f0 msgr2=0x7fdc5c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 -- 172.21.15.151:0/1045126339 shutdown_connections 2023-09-30T02:01:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.788+0000 7fdc61d10700 1 -- 172.21.15.151:0/1045126339 wait complete. 2023-09-30T02:01:01.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 Processor -- start 2023-09-30T02:01:01.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- start start 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 0x7fdc5c0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 0x7fdc5c139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c1364e0 0x7fdc5c137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0a5e40 con 0x7fdc5c06e660 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c137720 con 0x7fdc5c1364e0 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c13a350 con 0x7fdc5c067ab0 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc60d0e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 0x7fdc5c0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:01.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 0x7fdc5c139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc60d0e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 0x7fdc5c0a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:39156/0 (socket says 172.21.15.151:39156) 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc60d0e700 1 -- 172.21.15.151:0/3298826976 learned_addr learned my addr 172.21.15.151:0/3298826976 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc6150f700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c1364e0 0x7fdc5c137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 -- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 msgr2=0x7fdc5c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 0x7fdc5c0a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 -- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c1364e0 msgr2=0x7fdc5c137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c1364e0 0x7fdc5c137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 -- 172.21.15.151:0/3298826976 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c009420 con 0x7fdc5c06e660 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc5bfff700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 0x7fdc5c139e10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c016000 tx=0x7fdc4c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00bb20 con 0x7fdc5c06e660 2023-09-30T02:01:01.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c000d10 con 0x7fdc5c06e660 2023-09-30T02:01:01.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c011670 con 0x7fdc5c06e660 2023-09-30T02:01:01.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5c138160 con 0x7fdc5c06e660 2023-09-30T02:01:01.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.792+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c1385d0 con 0x7fdc5c06e660 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5c137580 con 0x7fdc5c06e660 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdc4c011aa0 con 0x7fdc5c06e660 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc58ff9700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 0x7fdc3c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fdc4c07de50 con 0x7fdc5c06e660 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc60d0e700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 0x7fdc3c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:01.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.796+0000 7fdc60d0e700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 0x7fdc3c062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc44007fd0 tx=0x7fdc44007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:01.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.804+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc5c137580 con 0x7fdc5c06e660 2023-09-30T02:01:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:01.996+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc5c137580 con 0x7fdc5c06e660 2023-09-30T02:01:02.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc58ff9700 1 -- 172.21.15.151:0/3298826976 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdc5c137580 con 0x7fdc5c06e660 2023-09-30T02:01:02.003 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:02.003 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:02.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 msgr2=0x7fdc3c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 0x7fdc3c062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc44007fd0 tx=0x7fdc44007e90 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 msgr2=0x7fdc5c139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 0x7fdc5c139e10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c016000 tx=0x7fdc4c017000 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 shutdown_connections 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdc3c060550 0x7fdc3c062a00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc5c1364e0 0x7fdc5c137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdc5c06e660 0x7fdc5c139e10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 --2- 172.21.15.151:0/3298826976 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdc5c067ab0 0x7fdc5c0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 >> 172.21.15.151:0/3298826976 conn(0x7fdc5c0b03f0 msgr2=0x7fdc5c0af800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 shutdown_connections 2023-09-30T02:01:02.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:02.000+0000 7fdc61d10700 1 -- 172.21.15.151:0/3298826976 wait complete. 2023-09-30T02:01:05.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 Processor -- start 2023-09-30T02:01:05.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 -- start start 2023-09-30T02:01:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00100b0070 0x7f001006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f001006e070 0x7f001013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.134 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f00100b0440 0x7f001006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00100abe30 con 0x7f00100b0440 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001006d690 con 0x7f001006e070 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0017c6f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001006d970 con 0x7f00100b0070 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f00100b0440 0x7f001006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f0015a0b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00100b0070 0x7f001006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f00100b0440 0x7f001006bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42106/0 (socket says 172.21.15.151:42106) 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 -- 172.21.15.151:0/3098327350 learned_addr learned my addr 172.21.15.151:0/3098327350 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001520a700 1 --2- 172.21.15.151:0/3098327350 >> v2:172.21.15.172:3300/0 conn(0x7f001006e070 0x7f001013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 -- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00100b0070 msgr2=0x7f001006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 --2- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00100b0070 0x7f001006ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 -- 172.21.15.151:0/3098327350 >> v2:172.21.15.172:3300/0 conn(0x7f001006e070 msgr2=0x7f001013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:05.135 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 --2- 172.21.15.151:0/3098327350 >> v2:172.21.15.172:3300/0 conn(0x7f001006e070 0x7f001013a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 -- 172.21.15.151:0/3098327350 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001006dc50 con 0x7f00100b0440 2023-09-30T02:01:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.127+0000 7f001620c700 1 --2- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f001006bf60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f001006a610 tx=0x7f0008009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a3e075ab3cdf2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0014a09700 1 -- 172.21.15.151:0/3098327350 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800b640 con 0x7f00100b0440 2023-09-30T02:01:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0014a09700 1 -- 172.21.15.151:0/3098327350 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000800b7a0 con 0x7f00100b0440 2023-09-30T02:01:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0014a09700 1 -- 172.21.15.151:0/3098327350 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800bb60 con 0x7f00100b0440 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 msgr2=0x7f001006bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 --2- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f001006bf60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f001006a610 tx=0x7f0008009250 comp rx=0 tx=0).stop 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- 172.21.15.151:0/3098327350 shutdown_connections 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 --2- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f001006bf60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 --2- 172.21.15.151:0/3098327350 >> v2:172.21.15.172:3300/0 conn(0x7f001006e070 0x7f001013a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 --2- 172.21.15.151:0/3098327350 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00100b0070 0x7f001006ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- 172.21.15.151:0/3098327350 >> 172.21.15.151:0/3098327350 conn(0x7f0010070fe0 msgr2=0x7f00100af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- 172.21.15.151:0/3098327350 shutdown_connections 2023-09-30T02:01:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- 172.21.15.151:0/3098327350 wait complete. 2023-09-30T02:01:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 Processor -- start 2023-09-30T02:01:05.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 -- start start 2023-09-30T02:01:05.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.131+0000 7f0017c6f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00100b0070 0x7f00100a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f0010134de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010134950 con 0x7f00100b0440 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100a50f0 con 0x7f00100b0070 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100a5d40 con 0x7f001006e070 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:39172/0 (socket says 172.21.15.151:39172) 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 -- 172.21.15.151:0/2113035815 learned_addr learned my addr 172.21.15.151:0/2113035815 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f001620c700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f0010134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f001520a700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00100b0070 0x7f00100a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 -- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00100b0070 msgr2=0x7f00100a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00100b0070 0x7f00100a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 -- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 msgr2=0x7f0010134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f0010134de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 -- 172.21.15.151:0/2113035815 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080094f0 con 0x7f001006e070 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0015a0b700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f000000ec80 tx=0x7f00000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000c850 con 0x7f001006e070 2023-09-30T02:01:05.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000000c9b0 con 0x7f001006e070 2023-09-30T02:01:05.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00100a6020 con 0x7f001006e070 2023-09-30T02:01:05.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000cd50 con 0x7f001006e070 2023-09-30T02:01:05.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010135540 con 0x7f001006e070 2023-09-30T02:01:05.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00100670a0 con 0x7f001006e070 2023-09-30T02:01:05.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.135+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0000013680 con 0x7f001006e070 2023-09-30T02:01:05.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.139+0000 7f0005ffb700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 0x7efff4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:05.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.139+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f000007c940 con 0x7f001006e070 2023-09-30T02:01:05.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.139+0000 7f001520a700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 0x7efff4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:05.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.139+0000 7f001520a700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 0x7efff4062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f00100abd20 tx=0x7f000c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:05.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.143+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f000004d120 con 0x7f001006e070 2023-09-30T02:01:05.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.343+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f001006e4e0 con 0x7f001006e070 2023-09-30T02:01:05.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.347+0000 7f0005ffb700 1 -- 172.21.15.151:0/2113035815 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0000050740 con 0x7f001006e070 2023-09-30T02:01:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:05.351 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 msgr2=0x7efff4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 0x7efff4062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f00100abd20 tx=0x7f000c00b040 comp rx=0 tx=0).stop 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 msgr2=0x7f00100a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f000000ec80 tx=0x7f00000091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 shutdown_connections 2023-09-30T02:01:05.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efff4060550 0x7efff4062a00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00100b0440 0x7f0010134de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f00100b0070 0x7f00100a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 --2- 172.21.15.151:0/2113035815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f001006e070 0x7f00100a4670 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 >> 172.21.15.151:0/2113035815 conn(0x7f0010070fe0 msgr2=0x7f00100acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 shutdown_connections 2023-09-30T02:01:05.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:05.351+0000 7f0017c6f700 1 -- 172.21.15.151:0/2113035815 wait complete. 2023-09-30T02:01:08.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:08.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 Processor -- start 2023-09-30T02:01:08.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 -- start start 2023-09-30T02:01:08.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fea0c0abf60 0x7fea0c06b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:08.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0c06cbe0 con 0x7fea0c0b02a0 2023-09-30T02:01:08.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0c06b080 con 0x7fea0c06e2b0 2023-09-30T02:01:08.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.479+0000 7fea13fc9700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0c0b0670 con 0x7fea0c0abf60 2023-09-30T02:01:08.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11d65700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fea0c0b02a0 0x7fea0c141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:41364/0 (socket says 172.21.15.151:41364) 2023-09-30T02:01:08.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11d65700 1 -- 172.21.15.151:0/1711700038 learned_addr learned my addr 172.21.15.151:0/1711700038 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:08.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11564700 1 --2- 172.21.15.151:0/1711700038 >> v2:172.21.15.172:3300/0 conn(0x7fea0c0abf60 0x7fea0c06b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924027306 0 0) 0x7fea0c06cbe0 con 0x7fea0c0b02a0 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8003570 con 0x7fea0c0b02a0 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11564700 1 -- 172.21.15.151:0/1711700038 >> v1:172.21.15.172:6789/0 conn(0x7fea0c06e2b0 legacy=0x7fea0c06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11564700 1 -- 172.21.15.151:0/1711700038 >> v1:172.21.15.151:6789/0 conn(0x7fea0c0b02a0 legacy=0x7fea0c141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103897539 0 0) 0x7fe9f8003570 con 0x7fea0c0b02a0 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11564700 1 -- 172.21.15.151:0/1711700038 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c0a9cb0 con 0x7fea0c0abf60 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea11564700 1 --2- 172.21.15.151:0/1711700038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c0abf60 0x7fea0c06b5a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea080098e0 tx=0x7fea080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee9373926ae7f9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:08.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00001470 con 0x7fea0c0abf60 2023-09-30T02:01:08.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08011420 con 0x7fea0c0abf60 2023-09-30T02:01:08.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea10d63700 1 -- 172.21.15.151:0/1711700038 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea080117c0 con 0x7fea0c0abf60 2023-09-30T02:01:08.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- 172.21.15.151:0/1711700038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c0abf60 msgr2=0x7fea0c06b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 --2- 172.21.15.151:0/1711700038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c0abf60 0x7fea0c06b5a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea080098e0 tx=0x7fea080091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- 172.21.15.151:0/1711700038 shutdown_connections 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 --2- 172.21.15.151:0/1711700038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c0abf60 0x7fea0c06b5a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- 172.21.15.151:0/1711700038 >> 172.21.15.151:0/1711700038 conn(0x7fea0c070fe0 msgr2=0x7fea0c0af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- 172.21.15.151:0/1711700038 shutdown_connections 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- 172.21.15.151:0/1711700038 wait complete. 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 Processor -- start 2023-09-30T02:01:08.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 -- start start 2023-09-30T02:01:08.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 0x7fea0c0a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.483+0000 7fea13fc9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 0x7fea0c1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0a4f30 con 0x7fea0c0abf60 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0a6150 con 0x7fea0c06e2b0 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c144800 con 0x7fea0c0b02a0 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea12566700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 0x7fea0c1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42138/0 (socket says 172.21.15.151:42138) 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 -- 172.21.15.151:0/3012022083 learned_addr learned my addr 172.21.15.151:0/3012022083 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11d65700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 0x7fea0c0a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 -- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 msgr2=0x7fea0c1517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 0x7fea0c1517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 -- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 msgr2=0x7fea0c0a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 0x7fea0c0a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 -- 172.21.15.151:0/3012022083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08009420 con 0x7fea0c0abf60 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea12566700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 0x7fea0c1517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11d65700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 0x7fea0c0a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:08.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea11564700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fea08002000 tx=0x7fea08016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0801caf0 con 0x7fea0c0abf60 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0801cc70 con 0x7fea0c0abf60 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0801a450 con 0x7fea0c0abf60 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c14ec30 con 0x7fea0c0abf60 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c151f50 con 0x7fea0c0abf60 2023-09-30T02:01:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.487+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c0670a0 con 0x7fea0c0abf60 2023-09-30T02:01:08.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.491+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fea0801a800 con 0x7fea0c0abf60 2023-09-30T02:01:08.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.491+0000 7fe9fdffb700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 0x7fe9f0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:08.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.491+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fea0807e0e0 con 0x7fea0c0abf60 2023-09-30T02:01:08.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.491+0000 7fea11d65700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 0x7fe9f0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:08.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.491+0000 7fea11d65700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 0x7fe9f0062a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea00007fd0 tx=0x7fea00000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:08.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.499+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fea0804e8c0 con 0x7fea0c0abf60 2023-09-30T02:01:08.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.687+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fea0c06c890 con 0x7fea0c0abf60 2023-09-30T02:01:08.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.687+0000 7fe9fdffb700 1 -- 172.21.15.151:0/3012022083 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fea08051ee0 con 0x7fea0c0abf60 2023-09-30T02:01:08.693 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:08.693 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:08.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 msgr2=0x7fe9f0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 0x7fe9f0062a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea00007fd0 tx=0x7fea00000bc0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 msgr2=0x7fea0c1512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fea08002000 tx=0x7fea08016000 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 shutdown_connections 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe9f0060550 0x7fe9f0062a00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea0c0b02a0 0x7fea0c1517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea0c0abf60 0x7fea0c1512b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 --2- 172.21.15.151:0/3012022083 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea0c06e2b0 0x7fea0c0a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 >> 172.21.15.151:0/3012022083 conn(0x7fea0c070fe0 msgr2=0x7fea0c0acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 shutdown_connections 2023-09-30T02:01:08.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:08.691+0000 7fea13fc9700 1 -- 172.21.15.151:0/3012022083 wait complete. 2023-09-30T02:01:11.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 Processor -- start 2023-09-30T02:01:11.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 -- start start 2023-09-30T02:01:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9d38140ac0 0x7f9d38140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3806c4b0 con 0x7f9d380b3f60 2023-09-30T02:01:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806c720 con 0x7f9d38140ac0 2023-09-30T02:01:11.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d37fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3806ca00 con 0x7f9d380b3b90 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:48942/0 (socket says 172.21.15.151:48942) 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1004053161 learned_addr learned my addr 172.21.15.151:0/1004053161 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d377fe700 1 --2- 172.21.15.151:0/1004053161 >> v2:172.21.15.172:3300/0 conn(0x7f9d38140ac0 0x7f9d38140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1004053161 >> v2:172.21.15.172:3300/0 conn(0x7f9d38140ac0 msgr2=0x7f9d38140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 --2- 172.21.15.151:0/1004053161 >> v2:172.21.15.172:3300/0 conn(0x7f9d38140ac0 0x7f9d38140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1004053161 >> v1:172.21.15.172:6789/0 conn(0x7f9d380b3f60 legacy=0x7f9d3806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1004053161 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3806b810 con 0x7f9d380b3b90 2023-09-30T02:01:11.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d36ffd700 1 --2- 172.21.15.151:0/1004053161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9d200099b0 tx=0x7f9d20009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5060ae4fb5266a86 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:11.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d35ffb700 1 -- 172.21.15.151:0/1004053161 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20009ea0 con 0x7f9d380b3b90 2023-09-30T02:01:11.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d35ffb700 1 -- 172.21.15.151:0/1004053161 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2000b860 con 0x7f9d380b3b90 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.823+0000 7f9d35ffb700 1 -- 172.21.15.151:0/1004053161 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000bc20 con 0x7f9d380b3b90 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- 172.21.15.151:0/1004053161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 msgr2=0x7f9d3806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1004053161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9d200099b0 tx=0x7f9d20009250 comp rx=0 tx=0).stop 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- 172.21.15.151:0/1004053161 shutdown_connections 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1004053161 >> v2:172.21.15.172:3300/0 conn(0x7f9d38140ac0 0x7f9d38140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1004053161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3b90 0x7f9d3806d100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- 172.21.15.151:0/1004053161 >> 172.21.15.151:0/1004053161 conn(0x7f9d38063990 msgr2=0x7f9d38070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- 172.21.15.151:0/1004053161 shutdown_connections 2023-09-30T02:01:11.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- 172.21.15.151:0/1004053161 wait complete. 2023-09-30T02:01:11.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 Processor -- start 2023-09-30T02:01:11.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.827+0000 7f9d37fff700 1 -- start start 2023-09-30T02:01:11.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 0x7f9d3814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38140ac0 0x7f9d3811b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3811dfd0 con 0x7f9d380b3b90 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3811dcd0 con 0x7f9d38140ac0 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3811f820 con 0x7f9d380b3f60 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57730/0 (socket says 172.21.15.151:57730) 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1422264145 learned_addr learned my addr 172.21.15.151:0/1422264145 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d367fc700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 0x7f9d3814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d377fe700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38140ac0 0x7f9d3811b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 msgr2=0x7f9d3814d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 0x7f9d3814d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38140ac0 msgr2=0x7f9d3811b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38140ac0 0x7f9d3811b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 -- 172.21.15.151:0/1422264145 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200094f0 con 0x7f9d380b3b90 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d367fc700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 0x7f9d3814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d36ffd700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d20006fd0 tx=0x7f9d20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2001f070 con 0x7f9d380b3b90 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20000bc0 con 0x7f9d380b3b90 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380ac780 con 0x7f9d380b3b90 2023-09-30T02:01:11.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000bc00 con 0x7f9d380b3b90 2023-09-30T02:01:11.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3811d8a0 con 0x7f9d380b3b90 2023-09-30T02:01:11.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d380008d0 con 0x7f9d380b3b90 2023-09-30T02:01:11.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.831+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9d20023070 con 0x7f9d380b3b90 2023-09-30T02:01:11.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.835+0000 7f9d2effd700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 0x7f9d18062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:11.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.835+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9d2007d8a0 con 0x7f9d380b3b90 2023-09-30T02:01:11.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.835+0000 7f9d367fc700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 0x7f9d18062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:11.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.835+0000 7f9d367fc700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 0x7f9d18062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d380ab880 tx=0x7f9d28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:11.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:11.839+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d2004e080 con 0x7f9d380b3b90 2023-09-30T02:01:12.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9d380adbd0 con 0x7f9d380b3b90 2023-09-30T02:01:12.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d2effd700 1 -- 172.21.15.151:0/1422264145 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9d20009300 con 0x7f9d380b3b90 2023-09-30T02:01:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:12.040 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 msgr2=0x7f9d18062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 0x7f9d18062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d380ab880 tx=0x7f9d28009040 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 msgr2=0x7f9d380abae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.035+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d20006fd0 tx=0x7f9d20017000 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 shutdown_connections 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f9d18060550 0x7f9d18062a00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38140ac0 0x7f9d3811b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d380b3f60 0x7f9d3814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 --2- 172.21.15.151:0/1422264145 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d380b3b90 0x7f9d380abae0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:12.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 >> 172.21.15.151:0/1422264145 conn(0x7f9d38063990 msgr2=0x7f9d3813eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:12.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 shutdown_connections 2023-09-30T02:01:12.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:12.039+0000 7f9d37fff700 1 -- 172.21.15.151:0/1422264145 wait complete. 2023-09-30T02:01:15.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 Processor -- start 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 -- start start 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c06c690 con 0x7fe31c0b3f80 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c06c970 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0b0100 con 0x7fe31c06d780 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe319d9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe31c0b3f80 0x7fe31c145290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58332/0 (socket says 172.21.15.151:58332) 2023-09-30T02:01:15.172 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe319d9b700 1 -- 172.21.15.151:0/1187639651 learned_addr learned my addr 172.21.15.151:0/1187639651 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:15.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31ad9d700 1 --2- 172.21.15.151:0/1187639651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753703025 0 0) 0x7fe31c06c690 con 0x7fe31c0b3f80 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003570 con 0x7fe31c0b3f80 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31ad9d700 1 -- 172.21.15.151:0/1187639651 >> v1:172.21.15.172:6789/0 conn(0x7fe31c0b3bb0 legacy=0x7fe31c06d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205544219 0 0) 0x7fe304003570 con 0x7fe31c0b3f80 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31ad9d700 1 -- 172.21.15.151:0/1187639651 >> v1:172.21.15.151:6789/0 conn(0x7fe31c0b3f80 legacy=0x7fe31c145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31ad9d700 1 -- 172.21.15.151:0/1187639651 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c06cc50 con 0x7fe31c06d780 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31ad9d700 1 --2- 172.21.15.151:0/1187639651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c1459a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c571a6763e7651 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3140017f0 con 0x7fe31c06d780 2023-09-30T02:01:15.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c011420 con 0x7fe31c06d780 2023-09-30T02:01:15.175 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.167+0000 7fe31959a700 1 -- 172.21.15.151:0/1187639651 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0117a0 con 0x7fe31c06d780 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- 172.21.15.151:0/1187639651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 msgr2=0x7fe31c1459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 --2- 172.21.15.151:0/1187639651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c1459a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- 172.21.15.151:0/1187639651 shutdown_connections 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 --2- 172.21.15.151:0/1187639651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c1459a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- 172.21.15.151:0/1187639651 >> 172.21.15.151:0/1187639651 conn(0x7fe31c063990 msgr2=0x7fe31c070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- 172.21.15.151:0/1187639651 shutdown_connections 2023-09-30T02:01:15.176 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- 172.21.15.151:0/1187639651 wait complete. 2023-09-30T02:01:15.177 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 Processor -- start 2023-09-30T02:01:15.177 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- start start 2023-09-30T02:01:15.178 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c089240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe31c0b3f80 0x7fe31c154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c154a40 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c154640 con 0x7fe31c0b3f80 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.171+0000 7fe31b59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c1547b0 con 0x7fe31c06d780 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57746/0 (socket says 172.21.15.151:57746) 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 -- 172.21.15.151:0/3986190422 learned_addr learned my addr 172.21.15.151:0/3986190422 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:15.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe31a59c700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c089240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:15.180 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe31ad9d700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe31c0b3f80 0x7fe31c154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 -- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 msgr2=0x7fe31c089240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c089240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 -- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe31c0b3f80 msgr2=0x7fe31c154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe31c0b3f80 0x7fe31c154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 -- 172.21.15.151:0/3986190422 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009420 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe319d9b700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe31400cbc0 tx=0x7fe314007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314004dc0 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31400a5d0 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c155470 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31400a990 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c08a200 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.182 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c0008d0 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.183 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.175+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe31400ad00 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.183 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.179+0000 7fe30a7fc700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 0x7fe2fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:15.183 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.179+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe31407bf60 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.183 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.179+0000 7fe31a59c700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 0x7fe2fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:15.184 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.179+0000 7fe31a59c700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 0x7fe2fc062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe310000f60 tx=0x7fe310007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:15.190 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.183+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe31404c7c0 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.379+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe31c0b4980 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.379+0000 7fe30a7fc700 1 -- 172.21.15.151:0/3986190422 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe31404fde0 con 0x7fe31c0b3bb0 2023-09-30T02:01:15.386 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:15.386 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 msgr2=0x7fe2fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 0x7fe2fc062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe310000f60 tx=0x7fe310007ea0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 msgr2=0x7fe31c086f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe31400cbc0 tx=0x7fe314007ca0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 shutdown_connections 2023-09-30T02:01:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe2fc060550 0x7fe2fc062a00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe31c0b3f80 0x7fe31c154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe31c0b3bb0 0x7fe31c086f60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 --2- 172.21.15.151:0/3986190422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe31c06d780 0x7fe31c089240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 >> 172.21.15.151:0/3986190422 conn(0x7fe31c063990 msgr2=0x7fe31c06d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 shutdown_connections 2023-09-30T02:01:15.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:15.383+0000 7fe31b59e700 1 -- 172.21.15.151:0/3986190422 wait complete. 2023-09-30T02:01:18.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 Processor -- start 2023-09-30T02:01:18.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 -- start start 2023-09-30T02:01:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 0x7f37900aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 0x7f379013b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3790138dc0 con 0x7f37900aaec0 2023-09-30T02:01:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37900abcb0 con 0x7f37900b0440 2023-09-30T02:01:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3796657700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37900abeb0 con 0x7f37900b0070 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f37877fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f37900aaec0 0x7f37900a8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58346/0 (socket says 172.21.15.151:58346) 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f37877fe700 1 -- 172.21.15.151:0/2119843794 learned_addr learned my addr 172.21.15.151:0/2119843794 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f378ffff700 1 --2- 172.21.15.151:0/2119843794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 0x7f37900aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.507+0000 7f3794bf4700 1 --2- 172.21.15.151:0/2119843794 >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 0x7f379013b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183852530 0 0) 0x7f3790138dc0 con 0x7f37900aaec0 2023-09-30T02:01:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c003570 con 0x7f37900aaec0 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378ffff700 1 -- 172.21.15.151:0/2119843794 >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 msgr2=0x7f379013b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690500179 0 0) 0x7f377c003570 con 0x7f37900aaec0 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378ffff700 1 --2- 172.21.15.151:0/2119843794 >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 0x7f379013b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378ffff700 1 -- 172.21.15.151:0/2119843794 >> v1:172.21.15.151:6789/0 conn(0x7f37900aaec0 legacy=0x7f37900a8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378ffff700 1 -- 172.21.15.151:0/2119843794 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900ac190 con 0x7f37900b0070 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3794bf4700 1 --2- 172.21.15.151:0/2119843794 >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 0x7f379013b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378ffff700 1 --2- 172.21.15.151:0/2119843794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 0x7f37900aa2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37800098e0 tx=0x7f37800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc616921ecf82bf6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778001470 con 0x7f37900b0070 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780011420 con 0x7f37900b0070 2023-09-30T02:01:18.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f378f7fe700 1 -- 172.21.15.151:0/2119843794 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37800117c0 con 0x7f37900b0070 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 -- 172.21.15.151:0/2119843794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 msgr2=0x7f37900aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 --2- 172.21.15.151:0/2119843794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 0x7f37900aa2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37800098e0 tx=0x7f37800091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 -- 172.21.15.151:0/2119843794 shutdown_connections 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 --2- 172.21.15.151:0/2119843794 >> v2:172.21.15.172:3300/0 conn(0x7f37900b0440 0x7f379013b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 --2- 172.21.15.151:0/2119843794 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900b0070 0x7f37900aa2d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 -- 172.21.15.151:0/2119843794 >> 172.21.15.151:0/2119843794 conn(0x7f3790070fe0 msgr2=0x7f37900af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 -- 172.21.15.151:0/2119843794 shutdown_connections 2023-09-30T02:01:18.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.511+0000 7f3796657700 1 -- 172.21.15.151:0/2119843794 wait complete. 2023-09-30T02:01:18.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 Processor -- start 2023-09-30T02:01:18.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 -- start start 2023-09-30T02:01:18.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37900b0070 0x7f379013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37900b0440 0x7f379014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790140d50 con 0x7f37900b0440 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37901179f0 con 0x7f37900b0070 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3796657700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379013cc40 con 0x7f37900aaec0 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f3794bf4700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37900b0440 0x7f379014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49006/0 (socket says 172.21.15.151:49006) 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 -- 172.21.15.151:0/1506879995 learned_addr learned my addr 172.21.15.151:0/1506879995 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:18.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f37877fe700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37900b0070 0x7f379013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 -- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37900b0070 msgr2=0x7f379013f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37900b0070 0x7f379013f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 -- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37900b0440 msgr2=0x7f379014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37900b0440 0x7f379014b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.515+0000 7f378ffff700 1 -- 172.21.15.151:0/1506879995 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3788009a20 con 0x7f37900aaec0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f378ffff700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3780009ed0 tx=0x7f3780016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780000d10 con 0x7f37900aaec0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378001cae0 con 0x7f37900aaec0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378001cea0 con 0x7f37900aaec0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3780009420 con 0x7f37900aaec0 2023-09-30T02:01:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379013eb10 con 0x7f37900aaec0 2023-09-30T02:01:18.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37900670a0 con 0x7f37900aaec0 2023-09-30T02:01:18.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.519+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f378001a660 con 0x7f37900aaec0 2023-09-30T02:01:18.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.523+0000 7f378cff9700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 0x7f3770062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:18.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.523+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f378007dca0 con 0x7f37900aaec0 2023-09-30T02:01:18.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.523+0000 7f37877fe700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 0x7f3770062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:18.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.523+0000 7f37877fe700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 0x7f3770062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37901415b0 tx=0x7f3778001aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:18.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.527+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f378004e480 con 0x7f37900aaec0 2023-09-30T02:01:18.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.723+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f37900aa9a0 con 0x7f37900aaec0 2023-09-30T02:01:18.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.723+0000 7f378cff9700 1 -- 172.21.15.151:0/1506879995 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3780029100 con 0x7f37900aaec0 2023-09-30T02:01:18.729 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:18.729 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 msgr2=0x7f3770062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 0x7f3770062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37901415b0 tx=0x7f3778001aa0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 msgr2=0x7f37901406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3780009ed0 tx=0x7f3780016000 comp rx=0 tx=0).stop 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 shutdown_connections 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3770060550 0x7f3770062a00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37900b0440 0x7f379014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37900b0070 0x7f379013f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 --2- 172.21.15.151:0/1506879995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37900aaec0 0x7f37901406d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 >> 172.21.15.151:0/1506879995 conn(0x7f3790070fe0 msgr2=0x7f3790137bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 shutdown_connections 2023-09-30T02:01:18.732 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:18.727+0000 7f3796657700 1 -- 172.21.15.151:0/1506879995 wait complete. 2023-09-30T02:01:21.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:21.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.851+0000 7fe9aa00e700 1 Processor -- start 2023-09-30T02:01:21.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.851+0000 7fe9aa00e700 1 -- start start 2023-09-30T02:01:21.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe9a414a3d0 0x7fe9a414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:21.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a40ac260 con 0x7fe9a40acd00 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a40b0e40 con 0x7fe9a40b0a70 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a406c5b0 con 0x7fe9a414a3d0 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a880b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe9a40acd00 0x7fe9a406db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:39578/0 (socket says 172.21.15.151:39578) 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a880b700 1 -- 172.21.15.151:0/2918372913 learned_addr learned my addr 172.21.15.151:0/2918372913 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a980d700 1 --2- 172.21.15.151:0/2918372913 >> v2:172.21.15.151:3300/0 conn(0x7fe9a414a3d0 0x7fe9a414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724811076 0 0) 0x7fe9a40ac260 con 0x7fe9a40acd00 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990003570 con 0x7fe9a40acd00 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217887454 0 0) 0x7fe9a40b0e40 con 0x7fe9a40b0a70 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a980d700 1 -- 172.21.15.151:0/2918372913 >> v1:172.21.15.172:6789/0 conn(0x7fe9a40b0a70 legacy=0x7fe9a406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a980d700 1 -- 172.21.15.151:0/2918372913 >> v1:172.21.15.151:6789/0 conn(0x7fe9a40acd00 legacy=0x7fe9a406db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a980d700 1 -- 172.21.15.151:0/2918372913 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a406c790 con 0x7fe9a414a3d0 2023-09-30T02:01:21.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9a980d700 1 --2- 172.21.15.151:0/2918372913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a414a3d0 0x7fe9a414c7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009e60 tx=0x7fe99c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fd63c42f4e78aa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:21.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c00e070 con 0x7fe9a414a3d0 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c00bb60 con 0x7fe9a414a3d0 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe99bfff700 1 -- 172.21.15.151:0/2918372913 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c012430 con 0x7fe9a414a3d0 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2918372913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a414a3d0 msgr2=0x7fe9a414c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2918372913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a414a3d0 0x7fe9a414c7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009e60 tx=0x7fe99c009780 comp rx=0 tx=0).stop 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2918372913 shutdown_connections 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2918372913 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a414a3d0 0x7fe9a414c7b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2918372913 >> 172.21.15.151:0/2918372913 conn(0x7fe9a4070510 msgr2=0x7fe9a4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2918372913 shutdown_connections 2023-09-30T02:01:21.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2918372913 wait complete. 2023-09-30T02:01:21.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 Processor -- start 2023-09-30T02:01:21.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.855+0000 7fe9aa00e700 1 -- start start 2023-09-30T02:01:21.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:21.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9a40b0a70 0x7fe9a4153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 0x7fe9a41544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4120e70 con 0x7fe9a40acd00 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40a8320 con 0x7fe9a414a3d0 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40a67f0 con 0x7fe9a40b0a70 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46158/0 (socket says 172.21.15.151:46158) 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 -- 172.21.15.151:0/2031104446 learned_addr learned my addr 172.21.15.151:0/2031104446 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a880b700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9a40b0a70 0x7fe9a4153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a980d700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 0x7fe9a41544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 -- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9a40b0a70 msgr2=0x7fe9a4153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9a40b0a70 0x7fe9a4153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 -- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 msgr2=0x7fe9a41544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 0x7fe9a41544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 -- 172.21.15.151:0/2031104446 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c009a20 con 0x7fe9a40acd00 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a980d700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 0x7fe9a41544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9a900c700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c7e0 tx=0x7fe9a0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0011590 con 0x7fe9a40acd00 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a0011710 con 0x7fe9a40acd00 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a40a7920 con 0x7fe9a40acd00 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0011ad0 con 0x7fe9a40acd00 2023-09-30T02:01:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a40a8070 con 0x7fe9a40acd00 2023-09-30T02:01:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.859+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a40a8560 con 0x7fe9a40acd00 2023-09-30T02:01:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.863+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe9a001b3b0 con 0x7fe9a40acd00 2023-09-30T02:01:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.863+0000 7fe9997fa700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 0x7fe988062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:21.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.863+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe9a007bbd0 con 0x7fe9a40acd00 2023-09-30T02:01:21.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.863+0000 7fe9a880b700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 0x7fe988062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:21.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.863+0000 7fe9a880b700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 0x7fe988062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4155000 tx=0x7fe994009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:21.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:21.871+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe9a40a8560 con 0x7fe9a40acd00 2023-09-30T02:01:22.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9a40a8560 con 0x7fe9a40acd00 2023-09-30T02:01:22.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9997fa700 1 -- 172.21.15.151:0/2031104446 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe9a40a8560 con 0x7fe9a40acd00 2023-09-30T02:01:22.075 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:22.075 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:22.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 msgr2=0x7fe988062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:22.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 0x7fe988062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4155000 tx=0x7fe994009040 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 msgr2=0x7fe9a41468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c7e0 tx=0x7fe9a0007df0 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 shutdown_connections 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe988060550 0x7fe988062a00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9a414a3d0 0x7fe9a41544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9a40b0a70 0x7fe9a4153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 --2- 172.21.15.151:0/2031104446 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9a40acd00 0x7fe9a41468f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:22.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 >> 172.21.15.151:0/2031104446 conn(0x7fe9a4070510 msgr2=0x7fe9a40ada90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:22.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 shutdown_connections 2023-09-30T02:01:22.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:22.071+0000 7fe9aa00e700 1 -- 172.21.15.151:0/2031104446 wait complete. 2023-09-30T02:01:25.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:25.205 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 Processor -- start 2023-09-30T02:01:25.205 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 -- start start 2023-09-30T02:01:25.206 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa654144bb0 0x7fa65413e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.206 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540abf80 con 0x7fa6540aab10 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa654145050 con 0x7fa6540675a0 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa653fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa654067970 con 0x7fa654144bb0 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa6527fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa6540aab10 0x7fa65413e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:39584/0 (socket says 172.21.15.151:39584) 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.199+0000 7fa6527fc700 1 -- 172.21.15.151:0/2465972474 learned_addr learned my addr 172.21.15.151:0/2465972474 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6537fe700 1 --2- 172.21.15.151:0/2465972474 >> v2:172.21.15.172:3300/0 conn(0x7fa654144bb0 0x7fa65413e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794971552 0 0) 0x7fa6540abf80 con 0x7fa6540aab10 2023-09-30T02:01:25.207 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa640003570 con 0x7fa6540aab10 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3405476368 0 0) 0x7fa640003570 con 0x7fa6540aab10 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 >> v2:172.21.15.172:3300/0 conn(0x7fa654144bb0 msgr2=0x7fa65413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 --2- 172.21.15.151:0/2465972474 >> v2:172.21.15.172:3300/0 conn(0x7fa654144bb0 0x7fa65413e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 >> v1:172.21.15.172:6789/0 conn(0x7fa6540675a0 legacy=0x7fa6540ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65406c620 con 0x7fa6540aab10 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7fa63c001470 con 0x7fa6540aab10 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 >> v1:172.21.15.151:6789/0 conn(0x7fa6540aab10 legacy=0x7fa65413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 0x7fa640003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 0x7fa640004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6400051c0 0x7fa64000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654067970 con 0x7fa6400051c0 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64000c160 con 0x7fa640004810 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64000c320 con 0x7fa640003a60 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 0x7fa640003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 0x7fa640004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6537fe700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6400051c0 0x7fa64000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa652ffd700 1 -- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 msgr2=0x7fa640003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:01:25.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa652ffd700 1 -- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 msgr2=0x7fa640003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 0x7fa640003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 0x7fa640003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 -- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 msgr2=0x7fa640003e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa640003a60 0x7fa640003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 -- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6400051c0 msgr2=0x7fa64000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6400051c0 0x7fa64000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 -- 172.21.15.151:0/2465972474 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640003570 con 0x7fa640004810 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6537fe700 1 -- 172.21.15.151:0/2465972474 reap_dead start 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 0x7fa640004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c3c0 tx=0x7fa63c001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=894510968576275a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:25.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00cc90 con 0x7fa640004810 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c00ce10 con 0x7fa640004810 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa651ffb700 1 -- 172.21.15.151:0/2465972474 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c005b70 con 0x7fa640004810 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 -- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 msgr2=0x7fa640004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 0x7fa640004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c3c0 tx=0x7fa63c001e50 comp rx=0 tx=0).stop 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 -- 172.21.15.151:0/2465972474 shutdown_connections 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 --2- 172.21.15.151:0/2465972474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa640004810 0x7fa640004c80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 -- 172.21.15.151:0/2465972474 >> 172.21.15.151:0/2465972474 conn(0x7fa6540b08c0 msgr2=0x7fa6540b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 -- 172.21.15.151:0/2465972474 shutdown_connections 2023-09-30T02:01:25.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.203+0000 7fa653fff700 1 -- 172.21.15.151:0/2465972474 wait complete. 2023-09-30T02:01:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 Processor -- start 2023-09-30T02:01:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- start start 2023-09-30T02:01:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 0x7fa6540a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6540aa050 0x7fa65413f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654143570 con 0x7fa640004810 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6541446e0 con 0x7fa654143070 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6541425c0 con 0x7fa6540aa050 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46180/0 (socket says 172.21.15.151:46180) 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 -- 172.21.15.151:0/2496189229 learned_addr learned my addr 172.21.15.151:0/2496189229 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa6537fe700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6540aa050 0x7fa65413f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 0x7fa6540a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 -- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6540aa050 msgr2=0x7fa65413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6540aa050 0x7fa65413f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 -- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 msgr2=0x7fa6540a9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 0x7fa6540a9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 -- 172.21.15.151:0/2496189229 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65413b420 con 0x7fa640004810 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 0x7fa6540a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa652ffd700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa644007a80 tx=0x7fa644007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400f850 con 0x7fa640004810 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400f9d0 con 0x7fa640004810 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6541428c0 con 0x7fa640004810 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400fdd0 con 0x7fa640004810 2023-09-30T02:01:25.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65413fd50 con 0x7fa640004810 2023-09-30T02:01:25.219 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.207+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6540008d0 con 0x7fa640004810 2023-09-30T02:01:25.219 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.211+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa644013600 con 0x7fa640004810 2023-09-30T02:01:25.219 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.211+0000 7fa64affd700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 0x7fa634062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:25.219 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.211+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fa64407c520 con 0x7fa640004810 2023-09-30T02:01:25.220 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.211+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 0x7fa634062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:25.220 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.211+0000 7fa6527fc700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 0x7fa634062a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa63c001e90 tx=0x7fa63c015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.219+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa64404cd00 con 0x7fa640004810 2023-09-30T02:01:25.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa65413f5f0 con 0x7fa640004810 2023-09-30T02:01:25.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa64affd700 1 -- 172.21.15.151:0/2496189229 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa644017100 con 0x7fa640004810 2023-09-30T02:01:25.420 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:25.420 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:25.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 msgr2=0x7fa634062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 0x7fa634062a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa63c001e90 tx=0x7fa63c015000 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 msgr2=0x7fa654142080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa644007a80 tx=0x7fa644007d30 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 shutdown_connections 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa634060550 0x7fa634062a00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa6540aa050 0x7fa65413f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.415+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa654143070 0x7fa6540a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.419+0000 7fa653fff700 1 --2- 172.21.15.151:0/2496189229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa640004810 0x7fa654142080 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.419+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 >> 172.21.15.151:0/2496189229 conn(0x7fa6540b08c0 msgr2=0x7fa65406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.419+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 shutdown_connections 2023-09-30T02:01:25.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:25.419+0000 7fa653fff700 1 -- 172.21.15.151:0/2496189229 wait complete. 2023-09-30T02:01:28.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:28.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.539+0000 7f70d1eec700 1 Processor -- start 2023-09-30T02:01:28.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.539+0000 7f70d1eec700 1 -- start start 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f70cc0aed00 0x7f70cc13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70cc13b2d0 con 0x7f70cc06d830 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70cc06d160 con 0x7f70cc0aed00 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc06d2a0 con 0x7f70cc0ae930 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44660/0 (socket says 172.21.15.151:44660) 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 -- 172.21.15.151:0/1057276080 learned_addr learned my addr 172.21.15.151:0/1057276080 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70c2ffd700 1 --2- 172.21.15.151:0/1057276080 >> v2:172.21.15.151:3300/0 conn(0x7f70cc0aed00 0x7f70cc13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 -- 172.21.15.151:0/1057276080 >> v2:172.21.15.151:3300/0 conn(0x7f70cc0aed00 msgr2=0x7f70cc13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:28.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 --2- 172.21.15.151:0/1057276080 >> v2:172.21.15.151:3300/0 conn(0x7f70cc0aed00 0x7f70cc13bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 -- 172.21.15.151:0/1057276080 >> v1:172.21.15.172:6789/0 conn(0x7f70cc06d830 legacy=0x7f70cc13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 -- 172.21.15.151:0/1057276080 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc13f3e0 con 0x7f70cc0ae930 2023-09-30T02:01:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70cb7fe700 1 --2- 172.21.15.151:0/1057276080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0099b0 tx=0x7f70bc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a67369bd2c7aff8f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70caffd700 1 -- 172.21.15.151:0/1057276080 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc009ea0 con 0x7f70cc0ae930 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70caffd700 1 -- 172.21.15.151:0/1057276080 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc00b860 con 0x7f70cc0ae930 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70caffd700 1 -- 172.21.15.151:0/1057276080 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc00bc00 con 0x7f70cc0ae930 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- 172.21.15.151:0/1057276080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 msgr2=0x7f70cc13b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1057276080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0099b0 tx=0x7f70bc009250 comp rx=0 tx=0).stop 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- 172.21.15.151:0/1057276080 shutdown_connections 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1057276080 >> v2:172.21.15.151:3300/0 conn(0x7f70cc0aed00 0x7f70cc13bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1057276080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc0ae930 0x7f70cc13b780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- 172.21.15.151:0/1057276080 >> 172.21.15.151:0/1057276080 conn(0x7f70cc070510 msgr2=0x7f70cc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:28.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- 172.21.15.151:0/1057276080 shutdown_connections 2023-09-30T02:01:28.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.543+0000 7f70d1eec700 1 -- 172.21.15.151:0/1057276080 wait complete. 2023-09-30T02:01:28.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 Processor -- start 2023-09-30T02:01:28.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 -- start start 2023-09-30T02:01:28.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 0x7f70cc0a7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70cc0aed00 0x7f70cc0aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0a9eb0 con 0x7f70cc0ae930 2023-09-30T02:01:28.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0a97b0 con 0x7f70cc0aed00 2023-09-30T02:01:28.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70d1eec700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0a9a90 con 0x7f70cc06d830 2023-09-30T02:01:28.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 0x7f70cc0a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70c2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 0x7f70cc0a7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44662/0 (socket says 172.21.15.151:44662) 2023-09-30T02:01:28.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70c2ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:46196/0 (socket says 172.21.15.151:46196) 2023-09-30T02:01:28.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70cb7fe700 1 -- 172.21.15.151:0/1692785074 learned_addr learned my addr 172.21.15.151:0/1692785074 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:28.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.547+0000 7f70cbfff700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70cc0aed00 0x7f70cc0aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 -- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 msgr2=0x7f70cc0a7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 0x7f70cc0a7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 -- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70cc0aed00 msgr2=0x7f70cc0aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70cc0aed00 0x7f70cc0aa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 -- 172.21.15.151:0/1692785074 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc0094f0 con 0x7f70cc0ae930 2023-09-30T02:01:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c2ffd700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e900 tx=0x7f70b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4013680 con 0x7f70cc0ae930 2023-09-30T02:01:28.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b40137e0 con 0x7f70cc0ae930 2023-09-30T02:01:28.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4013ba0 con 0x7f70cc0ae930 2023-09-30T02:01:28.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc0a67f0 con 0x7f70cc0ae930 2023-09-30T02:01:28.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc0ab010 con 0x7f70cc0ae930 2023-09-30T02:01:28.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc0ada90 con 0x7f70cc0ae930 2023-09-30T02:01:28.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f70b40123d0 con 0x7f70cc0ae930 2023-09-30T02:01:28.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.551+0000 7f70c3fff700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 0x7f70ac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:28.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.555+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f70b407c650 con 0x7f70cc0ae930 2023-09-30T02:01:28.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.555+0000 7f70cb7fe700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 0x7f70ac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:28.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.555+0000 7f70cb7fe700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 0x7f70ac062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009290 tx=0x7f70bc017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:28.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.559+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f70cc0ada90 con 0x7f70cc0ae930 2023-09-30T02:01:28.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.751+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70cc0ada90 con 0x7f70cc0ae930 2023-09-30T02:01:28.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.751+0000 7f70c3fff700 1 -- 172.21.15.151:0/1692785074 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f70cc0ada90 con 0x7f70cc0ae930 2023-09-30T02:01:28.758 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:28.758 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 msgr2=0x7f70ac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 0x7f70ac062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009290 tx=0x7f70bc017000 comp rx=0 tx=0).stop 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 msgr2=0x7f70cc0aa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e900 tx=0x7f70b40091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 shutdown_connections 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f70ac060550 0x7f70ac062a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70cc0aed00 0x7f70cc0aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70cc0ae930 0x7f70cc0aa320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 --2- 172.21.15.151:0/1692785074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70cc06d830 0x7f70cc0a7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 >> 172.21.15.151:0/1692785074 conn(0x7f70cc070510 msgr2=0x7f70cc0708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 shutdown_connections 2023-09-30T02:01:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:28.755+0000 7f70d1eec700 1 -- 172.21.15.151:0/1692785074 wait complete. 2023-09-30T02:01:31.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:31.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 Processor -- start 2023-09-30T02:01:31.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 -- start start 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 0x7f98e406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 0x7f98e4140f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e406c590 con 0x7f98e406b0f0 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e406c870 con 0x7f98e406b4c0 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e88b7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40b0a60 con 0x7f98e4140a80 2023-09-30T02:01:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e2d9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f98e406b0f0 0x7f98e406d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35526/0 (socket says 172.21.15.151:35526) 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e2d9d700 1 -- 172.21.15.151:0/4286265277 learned_addr learned my addr 172.21.15.151:0/4286265277 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e359e700 1 --2- 172.21.15.151:0/4286265277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 0x7f98e4140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e259c700 1 --2- 172.21.15.151:0/4286265277 >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 0x7f98e406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e1d9b700 1 -- 172.21.15.151:0/4286265277 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920715695 0 0) 0x7f98e406c590 con 0x7f98e406b0f0 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.879+0000 7f98e1d9b700 1 -- 172.21.15.151:0/4286265277 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98cc003570 con 0x7f98e406b0f0 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 -- 172.21.15.151:0/4286265277 >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 msgr2=0x7f98e406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 --2- 172.21.15.151:0/4286265277 >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 0x7f98e406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 -- 172.21.15.151:0/4286265277 >> v1:172.21.15.151:6789/0 conn(0x7f98e406b0f0 legacy=0x7f98e406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 -- 172.21.15.151:0/4286265277 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e406de70 con 0x7f98e4140a80 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e259c700 1 --2- 172.21.15.151:0/4286265277 >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 0x7f98e406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 --2- 172.21.15.151:0/4286265277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 0x7f98e4140f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009910 tx=0x7f98d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b750956439c065c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e1d9b700 1 -- 172.21.15.151:0/4286265277 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4011630 con 0x7f98e4140a80 2023-09-30T02:01:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e1d9b700 1 -- 172.21.15.151:0/4286265277 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d4011790 con 0x7f98e4140a80 2023-09-30T02:01:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e1d9b700 1 -- 172.21.15.151:0/4286265277 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4011b30 con 0x7f98e4140a80 2023-09-30T02:01:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- 172.21.15.151:0/4286265277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 msgr2=0x7f98e4140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- 172.21.15.151:0/4286265277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 0x7f98e4140f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009910 tx=0x7f98d40091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- 172.21.15.151:0/4286265277 shutdown_connections 2023-09-30T02:01:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- 172.21.15.151:0/4286265277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e4140a80 0x7f98e4140f00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- 172.21.15.151:0/4286265277 >> v2:172.21.15.172:3300/0 conn(0x7f98e406b4c0 0x7f98e406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- 172.21.15.151:0/4286265277 >> 172.21.15.151:0/4286265277 conn(0x7f98e4070510 msgr2=0x7f98e4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- 172.21.15.151:0/4286265277 shutdown_connections 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- 172.21.15.151:0/4286265277 wait complete. 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 Processor -- start 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- start start 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e406b0f0 0x7f98e411c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406b4c0 0x7f98e411c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e411cfe0 con 0x7f98e4140a80 2023-09-30T02:01:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40acc70 con 0x7f98e406b4c0 2023-09-30T02:01:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e88b7700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e413f4b0 con 0x7f98e406b0f0 2023-09-30T02:01:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e2d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e406b0f0 0x7f98e411c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59030/0 (socket says 172.21.15.151:59030) 2023-09-30T02:01:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 -- 172.21.15.151:0/1972937186 learned_addr learned my addr 172.21.15.151:0/1972937186 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e259c700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406b4c0 0x7f98e411c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.883+0000 7f98e359e700 1 -- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e406b0f0 msgr2=0x7f98e411c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e359e700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e406b0f0 0x7f98e411c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e359e700 1 -- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406b4c0 msgr2=0x7f98e411c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e359e700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406b4c0 0x7f98e411c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e359e700 1 -- 172.21.15.151:0/1972937186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009420 con 0x7f98e4140a80 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e359e700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f98e40b0ba0 tx=0x7f98d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40119d0 con 0x7f98e4140a80 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d4026430 con 0x7f98e4140a80 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d4009420 con 0x7f98e4140a80 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4011b30 con 0x7f98e4140a80 2023-09-30T02:01:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e40ab360 con 0x7f98e4140a80 2023-09-30T02:01:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e411d150 con 0x7f98e4140a80 2023-09-30T02:01:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.887+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f98d4023030 con 0x7f98e4140a80 2023-09-30T02:01:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.891+0000 7f98caffd700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 0x7f98d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.891+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f98d407de40 con 0x7f98e4140a80 2023-09-30T02:01:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.891+0000 7f98e2d9d700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 0x7f98d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.891+0000 7f98e2d9d700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 0x7f98d0062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005c50 tx=0x7f98dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:31.895+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f98d404e600 con 0x7f98e4140a80 2023-09-30T02:01:32.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.087+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98e40008d0 con 0x7f98e4140a80 2023-09-30T02:01:32.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.087+0000 7f98caffd700 1 -- 172.21.15.151:0/1972937186 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f98d4051c20 con 0x7f98e4140a80 2023-09-30T02:01:32.093 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:32.093 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 msgr2=0x7f98d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 0x7f98d0062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005c50 tx=0x7f98dc009040 comp rx=0 tx=0).stop 2023-09-30T02:01:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 msgr2=0x7f98e40aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f98e40b0ba0 tx=0x7f98d4017000 comp rx=0 tx=0).stop 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 shutdown_connections 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f98d0060550 0x7f98d0062a00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98e4140a80 0x7f98e40aae20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98e406b4c0 0x7f98e411c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 --2- 172.21.15.151:0/1972937186 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98e406b0f0 0x7f98e411c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 >> 172.21.15.151:0/1972937186 conn(0x7f98e4070510 msgr2=0x7f98e40b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 shutdown_connections 2023-09-30T02:01:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:32.091+0000 7f98e88b7700 1 -- 172.21.15.151:0/1972937186 wait complete. 2023-09-30T02:01:35.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:35.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.215+0000 7f95f45d0700 1 Processor -- start 2023-09-30T02:01:35.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.215+0000 7f95f45d0700 1 -- start start 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f95ec0708e0 0x7f95ec13e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f95ec0ac830 0x7f95ec137050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0a6d90 con 0x7f95ec070510 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0a7070 con 0x7f95ec0ac830 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec06ea10 con 0x7f95ec0708e0 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f95ec0ac830 0x7f95ec137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f95ec0ac830 0x7f95ec137050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59044/0 (socket says 172.21.15.151:59044) 2023-09-30T02:01:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 -- 172.21.15.151:0/618741600 learned_addr learned my addr 172.21.15.151:0/618741600 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f1b6b700 1 --2- 172.21.15.151:0/618741600 >> v2:172.21.15.172:3300/0 conn(0x7f95ec0708e0 0x7f95ec13e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 -- 172.21.15.151:0/618741600 >> v2:172.21.15.172:3300/0 conn(0x7f95ec0708e0 msgr2=0x7f95ec13e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/618741600 >> v2:172.21.15.172:3300/0 conn(0x7f95ec0708e0 0x7f95ec13e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 -- 172.21.15.151:0/618741600 >> v1:172.21.15.172:6789/0 conn(0x7f95ec070510 legacy=0x7f95ec13da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 -- 172.21.15.151:0/618741600 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec06ecf0 con 0x7f95ec0ac830 2023-09-30T02:01:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/618741600 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec137050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0a74c0 tx=0x7f95e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84748e67ee92b579 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f136a700 1 -- 172.21.15.151:0/618741600 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400e070 con 0x7f95ec0ac830 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f136a700 1 -- 172.21.15.151:0/618741600 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e400ba50 con 0x7f95ec0ac830 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f136a700 1 -- 172.21.15.151:0/618741600 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e40092e0 con 0x7f95ec0ac830 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- 172.21.15.151:0/618741600 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 msgr2=0x7f95ec137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 --2- 172.21.15.151:0/618741600 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec137050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0a74c0 tx=0x7f95e4009840 comp rx=0 tx=0).stop 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- 172.21.15.151:0/618741600 shutdown_connections 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 --2- 172.21.15.151:0/618741600 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec137050 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 --2- 172.21.15.151:0/618741600 >> v2:172.21.15.172:3300/0 conn(0x7f95ec0708e0 0x7f95ec13e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- 172.21.15.151:0/618741600 >> 172.21.15.151:0/618741600 conn(0x7f95ec067510 msgr2=0x7f95ec067910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- 172.21.15.151:0/618741600 shutdown_connections 2023-09-30T02:01:35.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.219+0000 7f95f45d0700 1 -- 172.21.15.151:0/618741600 wait complete. 2023-09-30T02:01:35.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 Processor -- start 2023-09-30T02:01:35.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 -- start start 2023-09-30T02:01:35.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95ec070510 0x7f95ec06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec0a8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec14ecc0 con 0x7f95ec0ac830 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0aa460 con 0x7f95ec070510 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f45d0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec14e940 con 0x7f95ec0708e0 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f1b6b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f1b6b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:44674/0 (socket says 172.21.15.151:44674) 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f1b6b700 1 -- 172.21.15.151:0/3151254339 learned_addr learned my addr 172.21.15.151:0/3151254339 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f236c700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95ec070510 0x7f95ec06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec0a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f2b6d700 1 -- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 msgr2=0x7f95ec14e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f2b6d700 1 -- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95ec070510 msgr2=0x7f95ec06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.223+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95ec070510 0x7f95ec06dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f2b6d700 1 -- 172.21.15.151:0/3151254339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4009ae0 con 0x7f95ec0ac830 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f1b6b700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f2b6d700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec0a8220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f95e4015000 tx=0x7f95e4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400be80 con 0x7f95ec0ac830 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e4007000 con 0x7f95ec0ac830 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec06e780 con 0x7f95ec0ac830 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e401c410 con 0x7f95ec0ac830 2023-09-30T02:01:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0a8980 con 0x7f95ec0ac830 2023-09-30T02:01:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec14ea80 con 0x7f95ec0ac830 2023-09-30T02:01:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.227+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f95e4003050 con 0x7f95ec0ac830 2023-09-30T02:01:35.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.231+0000 7f95e27fc700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 0x7f95d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.231+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f95e407de10 con 0x7f95ec0ac830 2023-09-30T02:01:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.231+0000 7f95f236c700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 0x7f95d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.231+0000 7f95f236c700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 0x7f95d0062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95e8003a80 tx=0x7f95e8007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:35.243 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.239+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f95e404e6a0 con 0x7f95ec0ac830 2023-09-30T02:01:35.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.439+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f95ec0670a0 con 0x7f95ec0ac830 2023-09-30T02:01:35.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95e27fc700 1 -- 172.21.15.151:0/3151254339 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f95e4051cc0 con 0x7f95ec0ac830 2023-09-30T02:01:35.446 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:35.446 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 msgr2=0x7f95d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 0x7f95d0062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95e8003a80 tx=0x7f95e8007ec0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 msgr2=0x7f95ec0a8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec0a8220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f95e4015000 tx=0x7f95e4018000 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 shutdown_connections 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f95d0060550 0x7f95d0062a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f95ec0ac830 0x7f95ec0a8220 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95ec0708e0 0x7f95ec14e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 --2- 172.21.15.151:0/3151254339 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95ec070510 0x7f95ec06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 >> 172.21.15.151:0/3151254339 conn(0x7f95ec067510 msgr2=0x7f95ec1354e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 shutdown_connections 2023-09-30T02:01:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:35.443+0000 7f95f45d0700 1 -- 172.21.15.151:0/3151254339 wait complete. 2023-09-30T02:01:38.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:38.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 Processor -- start 2023-09-30T02:01:38.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 -- start start 2023-09-30T02:01:38.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efed40b3b90 0x7efed406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 0x7efed4140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed406c4e0 con 0x7efed40b3f60 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed406c740 con 0x7efed40b3b90 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed981c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed406ca00 con 0x7efed4140ac0 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed3fff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efed40b3f60 0x7efed406d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35534/0 (socket says 172.21.15.151:35534) 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed3fff700 1 -- 172.21.15.151:0/3478255734 learned_addr learned my addr 172.21.15.151:0/3478255734 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 0x7efed4140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed881a700 1 --2- 172.21.15.151:0/3478255734 >> v2:172.21.15.172:3300/0 conn(0x7efed40b3b90 0x7efed406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526611083 0 0) 0x7efec4001490 con 0x7efed40b3f60 2023-09-30T02:01:38.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc003570 con 0x7efed40b3f60 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947205547 0 0) 0x7efebc003570 con 0x7efed40b3f60 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 -- 172.21.15.151:0/3478255734 >> v2:172.21.15.172:3300/0 conn(0x7efed40b3b90 msgr2=0x7efed406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 --2- 172.21.15.151:0/3478255734 >> v2:172.21.15.172:3300/0 conn(0x7efed40b3b90 0x7efed406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 -- 172.21.15.151:0/3478255734 >> v1:172.21.15.151:6789/0 conn(0x7efed40b3f60 legacy=0x7efed406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 -- 172.21.15.151:0/3478255734 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed406b810 con 0x7efed4140ac0 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed901b700 1 --2- 172.21.15.151:0/3478255734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 0x7efed4140f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efecc0098e0 tx=0x7efecc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd0e55b37c026f80 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4001980 con 0x7efed4140ac0 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc011420 con 0x7efed4140ac0 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.571+0000 7efed37fe700 1 -- 172.21.15.151:0/3478255734 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc0117c0 con 0x7efed4140ac0 2023-09-30T02:01:38.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- 172.21.15.151:0/3478255734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 msgr2=0x7efed4140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- 172.21.15.151:0/3478255734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 0x7efed4140f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efecc0098e0 tx=0x7efecc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- 172.21.15.151:0/3478255734 shutdown_connections 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- 172.21.15.151:0/3478255734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed4140ac0 0x7efed4140f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- 172.21.15.151:0/3478255734 >> v2:172.21.15.172:3300/0 conn(0x7efed40b3b90 0x7efed406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- 172.21.15.151:0/3478255734 >> 172.21.15.151:0/3478255734 conn(0x7efed4063990 msgr2=0x7efed4070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- 172.21.15.151:0/3478255734 shutdown_connections 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- 172.21.15.151:0/3478255734 wait complete. 2023-09-30T02:01:38.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 Processor -- start 2023-09-30T02:01:38.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- start start 2023-09-30T02:01:38.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 0x7efed411ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efed4140ac0 0x7efed414cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed414cb30 con 0x7efed40b3b90 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed411f170 con 0x7efed4140ac0 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed981c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed4121020 con 0x7efed40b3f60 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 0x7efed411ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59056/0 (socket says 172.21.15.151:59056) 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 -- 172.21.15.151:0/1070580563 learned_addr learned my addr 172.21.15.151:0/1070580563 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed901b700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efed4140ac0 0x7efed414cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 -- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 msgr2=0x7efed411ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 0x7efed411ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 -- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efed4140ac0 msgr2=0x7efed414cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efed4140ac0 0x7efed414cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed881a700 1 -- 172.21.15.151:0/1070580563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc009420 con 0x7efed40b3b90 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.575+0000 7efed3fff700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 0x7efed411ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed881a700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efec800e960 tx=0x7efec80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8013680 con 0x7efed40b3b90 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec80137e0 con 0x7efed40b3b90 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8013b80 con 0x7efed40b3b90 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed4121290 con 0x7efed40b3b90 2023-09-30T02:01:38.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed414d500 con 0x7efed40b3b90 2023-09-30T02:01:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed40008d0 con 0x7efed40b3b90 2023-09-30T02:01:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efec800c490 con 0x7efed40b3b90 2023-09-30T02:01:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 0x7efec0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:38.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efec807cb30 con 0x7efed40b3b90 2023-09-30T02:01:38.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.579+0000 7efed3fff700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 0x7efec0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:38.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.583+0000 7efed3fff700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 0x7efec0062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efed414a7d0 tx=0x7efec4001690 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:38.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.587+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efec804d270 con 0x7efed40b3b90 2023-09-30T02:01:38.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efed40b2e70 con 0x7efed40b3b90 2023-09-30T02:01:38.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed0ff9700 1 -- 172.21.15.151:0/1070580563 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efec8050890 con 0x7efed40b3b90 2023-09-30T02:01:38.784 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:38.784 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:38.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 msgr2=0x7efec0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 0x7efec0062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efed414a7d0 tx=0x7efec4001690 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 msgr2=0x7efed411f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efec800e960 tx=0x7efec80091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 shutdown_connections 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efec0060550 0x7efec0062a00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efed4140ac0 0x7efed414cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efed40b3f60 0x7efed411ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 --2- 172.21.15.151:0/1070580563 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efed40b3b90 0x7efed411f5f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.779+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 >> 172.21.15.151:0/1070580563 conn(0x7efed4063990 msgr2=0x7efed413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.783+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 shutdown_connections 2023-09-30T02:01:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:38.783+0000 7efed981c700 1 -- 172.21.15.151:0/1070580563 wait complete. 2023-09-30T02:01:41.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:41.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 Processor -- start 2023-09-30T02:01:41.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- start start 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9698140ac0 0x7f9698140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969806c4b0 con 0x7f96980b3b90 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969806c720 con 0x7f9698140ac0 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969806ca00 con 0x7f96980b3f60 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:60326/0 (socket says 172.21.15.151:60326) 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 -- 172.21.15.151:0/3381143296 learned_addr learned my addr 172.21.15.151:0/3381143296 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969df07700 1 --2- 172.21.15.151:0/3381143296 >> v2:172.21.15.151:3300/0 conn(0x7f9698140ac0 0x7f9698140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 -- 172.21.15.151:0/3381143296 >> v2:172.21.15.151:3300/0 conn(0x7f9698140ac0 msgr2=0x7f9698140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 --2- 172.21.15.151:0/3381143296 >> v2:172.21.15.151:3300/0 conn(0x7f9698140ac0 0x7f9698140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 -- 172.21.15.151:0/3381143296 >> v1:172.21.15.151:6789/0 conn(0x7f96980b3b90 legacy=0x7f969806d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 -- 172.21.15.151:0/3381143296 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969806b810 con 0x7f96980b3f60 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969cf05700 1 --2- 172.21.15.151:0/3381143296 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9690009f20 tx=0x7f9690009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0e9a71cb425ef4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f968bfff700 1 -- 172.21.15.151:0/3381143296 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969000e070 con 0x7f96980b3f60 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f968bfff700 1 -- 172.21.15.151:0/3381143296 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969000b9d0 con 0x7f96980b3f60 2023-09-30T02:01:41.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f968bfff700 1 -- 172.21.15.151:0/3381143296 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96900092e0 con 0x7f96980b3f60 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- 172.21.15.151:0/3381143296 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 msgr2=0x7f969806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 --2- 172.21.15.151:0/3381143296 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9690009f20 tx=0x7f9690009840 comp rx=0 tx=0).stop 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- 172.21.15.151:0/3381143296 shutdown_connections 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 --2- 172.21.15.151:0/3381143296 >> v2:172.21.15.151:3300/0 conn(0x7f9698140ac0 0x7f9698140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 --2- 172.21.15.151:0/3381143296 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3f60 0x7f969806d810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- 172.21.15.151:0/3381143296 >> 172.21.15.151:0/3381143296 conn(0x7f9698063990 msgr2=0x7f9698070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- 172.21.15.151:0/3381143296 shutdown_connections 2023-09-30T02:01:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.911+0000 7f969e708700 1 -- 172.21.15.151:0/3381143296 wait complete. 2023-09-30T02:01:41.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 Processor -- start 2023-09-30T02:01:41.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 -- start start 2023-09-30T02:01:41.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3b90 0x7f96981503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9698140ac0 0x7f969813e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969811f110 con 0x7f96980b3f60 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969811eda0 con 0x7f9698140ac0 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969e708700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969814a730 con 0x7f96980b3b90 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969cf05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969d706700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3b90 0x7f96981503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969cf05700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45214/0 (socket says 172.21.15.151:45214) 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.915+0000 7f969cf05700 1 -- 172.21.15.151:0/2684350387 learned_addr learned my addr 172.21.15.151:0/2684350387 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969df07700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9698140ac0 0x7f969813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 -- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3b90 msgr2=0x7f96981503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3b90 0x7f96981503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 -- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9698140ac0 msgr2=0x7f969813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9698140ac0 0x7f969813e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 -- 172.21.15.151:0/2684350387 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9690009ae0 con 0x7f96980b3f60 2023-09-30T02:01:41.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969cf05700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f969806cb40 tx=0x7f9690018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690007000 con 0x7f96980b3f60 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9690007790 con 0x7f96980b3f60 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969813e770 con 0x7f96980b3f60 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690007b90 con 0x7f96980b3f60 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9698150e40 con 0x7f96980b3f60 2023-09-30T02:01:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9698151060 con 0x7f96980b3f60 2023-09-30T02:01:41.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.919+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f969001d420 con 0x7f96980b3f60 2023-09-30T02:01:41.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.923+0000 7f96897fa700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 0x7f967c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:41.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.923+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f969007e350 con 0x7f96980b3f60 2023-09-30T02:01:41.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.923+0000 7f969d706700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 0x7f967c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:41.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.923+0000 7f969d706700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 0x7f967c062a00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f968c003950 tx=0x7f968c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:41.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:41.927+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9698151060 con 0x7f96980b3f60 2023-09-30T02:01:42.127 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9698151060 con 0x7f96980b3f60 2023-09-30T02:01:42.127 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f96897fa700 1 -- 172.21.15.151:0/2684350387 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9698151060 con 0x7f96980b3f60 2023-09-30T02:01:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:42.128 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:42.130 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 msgr2=0x7f967c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 0x7f967c062a00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f968c003950 tx=0x7f968c007ec0 comp rx=0 tx=0).stop 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 msgr2=0x7f969813dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f969806cb40 tx=0x7f9690018000 comp rx=0 tx=0).stop 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 shutdown_connections 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f967c060550 0x7f967c062a00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9698140ac0 0x7f969813e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:42.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96980b3f60 0x7f969813dcf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:42.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.123+0000 7f969e708700 1 --2- 172.21.15.151:0/2684350387 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96980b3b90 0x7f96981503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:42.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.127+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 >> 172.21.15.151:0/2684350387 conn(0x7f9698063990 msgr2=0x7f969813f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:42.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.127+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 shutdown_connections 2023-09-30T02:01:42.132 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:42.127+0000 7f969e708700 1 -- 172.21.15.151:0/2684350387 wait complete. 2023-09-30T02:01:45.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:45.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 Processor -- start 2023-09-30T02:01:45.251 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- start start 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f375c0abf60 0x7f375c06b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 0x7f375c06e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c06cbe0 con 0x7f375c0b02a0 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375c06b080 con 0x7f375c0abf60 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0b0670 con 0x7f375c06e2b0 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376210a700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f375c0b02a0 0x7f375c141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:51686/0 (socket says 172.21.15.151:51686) 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 0x7f375c06e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376210a700 1 -- 172.21.15.151:0/1982447333 learned_addr learned my addr 172.21.15.151:0/1982447333 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f3761909700 1 --2- 172.21.15.151:0/1982447333 >> v2:172.21.15.172:3300/0 conn(0x7f375c0abf60 0x7f375c06b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f3761108700 1 -- 172.21.15.151:0/1982447333 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635584208 0 0) 0x7f375c06cbe0 con 0x7f375c0b02a0 2023-09-30T02:01:45.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 -- 172.21.15.151:0/1982447333 >> v2:172.21.15.172:3300/0 conn(0x7f375c0abf60 msgr2=0x7f375c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:45.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 --2- 172.21.15.151:0/1982447333 >> v2:172.21.15.172:3300/0 conn(0x7f375c0abf60 0x7f375c06b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 -- 172.21.15.151:0/1982447333 >> v1:172.21.15.151:6789/0 conn(0x7f375c0b02a0 legacy=0x7f375c141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 -- 172.21.15.151:0/1982447333 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c0a9cb0 con 0x7f375c06e2b0 2023-09-30T02:01:45.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376290b700 1 --2- 172.21.15.151:0/1982447333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 0x7f375c06e750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f374c0098e0 tx=0x7f374c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d5e722c51ba2624 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f3761108700 1 -- 172.21.15.151:0/1982447333 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c00b640 con 0x7f375c06e2b0 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f3761108700 1 -- 172.21.15.151:0/1982447333 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374c00b7a0 con 0x7f375c06e2b0 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f3761108700 1 -- 172.21.15.151:0/1982447333 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c00bb40 con 0x7f375c06e2b0 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- 172.21.15.151:0/1982447333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 msgr2=0x7f375c06e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 --2- 172.21.15.151:0/1982447333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 0x7f375c06e750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f374c0098e0 tx=0x7f374c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- 172.21.15.151:0/1982447333 shutdown_connections 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 --2- 172.21.15.151:0/1982447333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c06e2b0 0x7f375c06e750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 --2- 172.21.15.151:0/1982447333 >> v2:172.21.15.172:3300/0 conn(0x7f375c0abf60 0x7f375c06b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- 172.21.15.151:0/1982447333 >> 172.21.15.151:0/1982447333 conn(0x7f375c070fe0 msgr2=0x7f375c0af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- 172.21.15.151:0/1982447333 shutdown_connections 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.247+0000 7f376436e700 1 -- 172.21.15.151:0/1982447333 wait complete. 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 Processor -- start 2023-09-30T02:01:45.256 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- start start 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375c06e2b0 0x7f375c154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 0x7f375c145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0a7170 con 0x7f375c0abf60 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c147380 con 0x7f375c06e2b0 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c1474f0 con 0x7f375c0b02a0 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376290b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 0x7f375c145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45228/0 (socket says 172.21.15.151:45228) 2023-09-30T02:01:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376290b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 0x7f375c145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:60348/0 (socket says 172.21.15.151:60348) 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 -- 172.21.15.151:0/3876815147 learned_addr learned my addr 172.21.15.151:0/3876815147 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376210a700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375c06e2b0 0x7f375c154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 -- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 msgr2=0x7f375c145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 0x7f375c145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 -- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375c06e2b0 msgr2=0x7f375c154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375c06e2b0 0x7f375c154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 -- 172.21.15.151:0/3876815147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c009420 con 0x7f375c0abf60 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f3761909700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3754009eb0 tx=0x7f375400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375400cb10 con 0x7f375c0abf60 2023-09-30T02:01:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375400f040 con 0x7f375c0abf60 2023-09-30T02:01:45.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c155380 con 0x7f375c0abf60 2023-09-30T02:01:45.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754014480 con 0x7f375c0abf60 2023-09-30T02:01:45.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.251+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c1461c0 con 0x7f375c0abf60 2023-09-30T02:01:45.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c0670a0 con 0x7f375c0abf60 2023-09-30T02:01:45.260 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f37540147f0 con 0x7f375c0abf60 2023-09-30T02:01:45.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f375a7fc700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 0x7f3748062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:45.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f375407d0f0 con 0x7f375c0abf60 2023-09-30T02:01:45.261 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f376210a700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 0x7f3748062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:45.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.255+0000 7f376210a700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 0x7f3748062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3750007160 tx=0x7f3750007f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:45.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.267+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f375404d8d0 con 0x7f375c0abf60 2023-09-30T02:01:45.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.459+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f375c06c890 con 0x7f375c0abf60 2023-09-30T02:01:45.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f375a7fc700 1 -- 172.21.15.151:0/3876815147 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3754050ef0 con 0x7f375c0abf60 2023-09-30T02:01:45.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:45.467 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:45.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 msgr2=0x7f3748062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 0x7f3748062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3750007160 tx=0x7f3750007f50 comp rx=0 tx=0).stop 2023-09-30T02:01:45.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 msgr2=0x7f375c154e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3754009eb0 tx=0x7f375400b040 comp rx=0 tx=0).stop 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 shutdown_connections 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3748060550 0x7f3748062a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f375c0b02a0 0x7f375c145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f375c0abf60 0x7f375c154e40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 --2- 172.21.15.151:0/3876815147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f375c06e2b0 0x7f375c154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 >> 172.21.15.151:0/3876815147 conn(0x7f375c070fe0 msgr2=0x7f375c0acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 shutdown_connections 2023-09-30T02:01:45.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:45.463+0000 7f376436e700 1 -- 172.21.15.151:0/3876815147 wait complete. 2023-09-30T02:01:48.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:48.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 Processor -- start 2023-09-30T02:01:48.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 -- start start 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 0x7f4a4c0aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4c138c00 con 0x7f4a4c13afd0 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4c0abbd0 con 0x7f4a4c0ac140 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a527df700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0abdd0 con 0x7f4a4c0ac510 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:60362/0 (socket says 172.21.15.151:60362) 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 -- 172.21.15.151:0/316486895 learned_addr learned my addr 172.21.15.151:0/316486895 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:48.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a517dd700 1 --2- 172.21.15.151:0/316486895 >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 0x7f4a4c0aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 -- 172.21.15.151:0/316486895 >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 msgr2=0x7f4a4c0aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 --2- 172.21.15.151:0/316486895 >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 0x7f4a4c0aa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 -- 172.21.15.151:0/316486895 >> v1:172.21.15.172:6789/0 conn(0x7f4a4c13afd0 legacy=0x7f4a4c0a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a50fdc700 1 -- 172.21.15.151:0/316486895 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c0a9b50 con 0x7f4a4c0ac510 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.587+0000 7f4a517dd700 1 --2- 172.21.15.151:0/316486895 >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 0x7f4a4c0aa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a50fdc700 1 --2- 172.21.15.151:0/316486895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009f20 tx=0x7f4a48009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8258c4881a3cb88f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a43fff700 1 -- 172.21.15.151:0/316486895 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800e070 con 0x7f4a4c0ac510 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a43fff700 1 -- 172.21.15.151:0/316486895 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4800ba40 con 0x7f4a4c0ac510 2023-09-30T02:01:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a43fff700 1 -- 172.21.15.151:0/316486895 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a480092e0 con 0x7f4a4c0ac510 2023-09-30T02:01:48.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 -- 172.21.15.151:0/316486895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 msgr2=0x7f4a4c06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:48.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 --2- 172.21.15.151:0/316486895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009f20 tx=0x7f4a48009840 comp rx=0 tx=0).stop 2023-09-30T02:01:48.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 -- 172.21.15.151:0/316486895 shutdown_connections 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 --2- 172.21.15.151:0/316486895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c0ac510 0x7f4a4c06e870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 --2- 172.21.15.151:0/316486895 >> v2:172.21.15.172:3300/0 conn(0x7f4a4c0ac140 0x7f4a4c0aa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 -- 172.21.15.151:0/316486895 >> 172.21.15.151:0/316486895 conn(0x7f4a4c070510 msgr2=0x7f4a4c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 -- 172.21.15.151:0/316486895 shutdown_connections 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.591+0000 7f4a527df700 1 -- 172.21.15.151:0/316486895 wait complete. 2023-09-30T02:01:48.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 Processor -- start 2023-09-30T02:01:48.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- start start 2023-09-30T02:01:48.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 0x7f4a4c0a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a4c0ac510 0x7f4a4c13e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 0x7f4a4c0a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0a8480 con 0x7f4a4c0ac140 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0a6420 con 0x7f4a4c0ac510 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c13f240 con 0x7f4a4c13afd0 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a517dd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 0x7f4a4c0a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a517dd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 0x7f4a4c0a5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45234/0 (socket says 172.21.15.151:45234) 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 0x7f4a4c0a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a517dd700 1 -- 172.21.15.151:0/3674188686 learned_addr learned my addr 172.21.15.151:0/3674188686 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:48.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a50fdc700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a4c0ac510 0x7f4a4c13e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 -- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a4c0ac510 msgr2=0x7f4a4c13e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a4c0ac510 0x7f4a4c13e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 -- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 msgr2=0x7f4a4c0a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 0x7f4a4c0a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 -- 172.21.15.151:0/3674188686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48009ae0 con 0x7f4a4c13afd0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a51fde700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 0x7f4a4c0a6990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400b440 tx=0x7f4a44006f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44011400 con 0x7f4a4c13afd0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a44011560 con 0x7f4a4c13afd0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c0a4570 con 0x7f4a4c13afd0 2023-09-30T02:01:48.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44011920 con 0x7f4a4c13afd0 2023-09-30T02:01:48.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.595+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c0a7070 con 0x7f4a4c13afd0 2023-09-30T02:01:48.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.599+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4a44011d10 con 0x7f4a4c13afd0 2023-09-30T02:01:48.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.599+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c0a65e0 con 0x7f4a4c13afd0 2023-09-30T02:01:48.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.599+0000 7f4a417fa700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:48.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.599+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4a4407bf60 con 0x7f4a4c13afd0 2023-09-30T02:01:48.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.599+0000 7f4a517dd700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:48.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.603+0000 7f4a517dd700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 0x7f4a30062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c006fd0 tx=0x7f4a3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:48.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.607+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4a4404c720 con 0x7f4a4c13afd0 2023-09-30T02:01:48.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.799+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4a4c0a82c0 con 0x7f4a4c13afd0 2023-09-30T02:01:48.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.799+0000 7f4a417fa700 1 -- 172.21.15.151:0/3674188686 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4a4404fd40 con 0x7f4a4c13afd0 2023-09-30T02:01:48.804 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:48.804 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 msgr2=0x7f4a30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 0x7f4a30062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c006fd0 tx=0x7f4a3c008040 comp rx=0 tx=0).stop 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 msgr2=0x7f4a4c0a6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 0x7f4a4c0a6990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400b440 tx=0x7f4a44006f60 comp rx=0 tx=0).stop 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 shutdown_connections 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a4c13afd0 0x7f4a4c0a6990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a4c0ac510 0x7f4a4c13e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 --2- 172.21.15.151:0/3674188686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a4c0ac140 0x7f4a4c0a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:48.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 >> 172.21.15.151:0/3674188686 conn(0x7f4a4c070510 msgr2=0x7f4a4c062d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:48.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 shutdown_connections 2023-09-30T02:01:48.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:48.803+0000 7f4a527df700 1 -- 172.21.15.151:0/3674188686 wait complete. 2023-09-30T02:01:51.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:51.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 Processor -- start 2023-09-30T02:01:51.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- start start 2023-09-30T02:01:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f56100ab220 0x7f56100ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 0x7f561013d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56100ad0e0 con 0x7f56100ab220 2023-09-30T02:01:51.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5610139ed0 con 0x7f561013aa00 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56100a9b00 con 0x7f56100ab5f0 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43224/0 (socket says 172.21.15.151:43224) 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 -- 172.21.15.151:0/424592734 learned_addr learned my addr 172.21.15.151:0/424592734 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560e59c700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.151:3300/0 conn(0x7f56100ab220 0x7f56100ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560ed9d700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 0x7f561013d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 -- 172.21.15.151:0/424592734 >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 msgr2=0x7f561013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 0x7f561013d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 -- 172.21.15.151:0/424592734 >> v2:172.21.15.151:3300/0 conn(0x7f56100ab220 msgr2=0x7f56100ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.151:3300/0 conn(0x7f56100ab220 0x7f56100ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 -- 172.21.15.151:0/424592734 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56100acbf0 con 0x7f56100ab5f0 2023-09-30T02:01:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560ed9d700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 0x7f561013d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T02:01:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560dd9b700 1 --2- 172.21.15.151:0/424592734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56100ad050 tx=0x7f5600009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=164a1d705e7d6c89 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560d59a700 1 -- 172.21.15.151:0/424592734 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600009ea0 con 0x7f56100ab5f0 2023-09-30T02:01:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560d59a700 1 -- 172.21.15.151:0/424592734 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560000b8b0 con 0x7f56100ab5f0 2023-09-30T02:01:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560d59a700 1 -- 172.21.15.151:0/424592734 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560000bc50 con 0x7f56100ab5f0 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- 172.21.15.151:0/424592734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 msgr2=0x7f561013a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- 172.21.15.151:0/424592734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56100ad050 tx=0x7f5600009250 comp rx=0 tx=0).stop 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- 172.21.15.151:0/424592734 shutdown_connections 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.172:3300/0 conn(0x7f561013aa00 0x7f561013d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- 172.21.15.151:0/424592734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561013a4c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 --2- 172.21.15.151:0/424592734 >> v2:172.21.15.151:3300/0 conn(0x7f56100ab220 0x7f56100ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- 172.21.15.151:0/424592734 >> 172.21.15.151:0/424592734 conn(0x7f5610070510 msgr2=0x7f5610070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- 172.21.15.151:0/424592734 shutdown_connections 2023-09-30T02:01:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.931+0000 7f560f59e700 1 -- 172.21.15.151:0/424592734 wait complete. 2023-09-30T02:01:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 Processor -- start 2023-09-30T02:01:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 -- start start 2023-09-30T02:01:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 0x7f56100b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561013aa00 0x7f56101369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610139c80 con 0x7f56100ab220 2023-09-30T02:01:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56100ae820 con 0x7f561013aa00 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610137cd0 con 0x7f56100ab5f0 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 0x7f56100b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43226/0 (socket says 172.21.15.151:43226) 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 -- 172.21.15.151:0/2975752585 learned_addr learned my addr 172.21.15.151:0/2975752585 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560ed9d700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561013aa00 0x7f56101369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 -- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561013aa00 msgr2=0x7f56101369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561013aa00 0x7f56101369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 -- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 msgr2=0x7f56100b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 0x7f56100b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:51.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 -- 172.21.15.151:0/2975752585 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000094f0 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560e59c700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 0x7f56100b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560dd9b700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56000094c0 tx=0x7f5600017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600000d10 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560000ba20 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.935+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610139040 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.939+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600025670 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.939+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56100adb40 con 0x7f56100ab5f0 2023-09-30T02:01:51.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.939+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56100008d0 con 0x7f56100ab5f0 2023-09-30T02:01:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.939+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5600023070 con 0x7f56100ab5f0 2023-09-30T02:01:51.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.943+0000 7f55fe7fc700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 0x7f55f0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:51.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.943+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f560007da00 con 0x7f56100ab5f0 2023-09-30T02:01:51.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.943+0000 7f560e59c700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 0x7f55f0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:51.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.943+0000 7f560e59c700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 0x7f55f0062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5604011480 tx=0x7f56040091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:51.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:51.947+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f560004e140 con 0x7f56100ab5f0 2023-09-30T02:01:52.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.143+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f561006e910 con 0x7f56100ab5f0 2023-09-30T02:01:52.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.143+0000 7f55fe7fc700 1 -- 172.21.15.151:0/2975752585 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5600051760 con 0x7f56100ab5f0 2023-09-30T02:01:52.149 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:52.149 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 msgr2=0x7f55f0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 0x7f55f0062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5604011480 tx=0x7f56040091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 msgr2=0x7f561012e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f56000094c0 tx=0x7f5600017000 comp rx=0 tx=0).stop 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 shutdown_connections 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f55f0060550 0x7f55f0062a00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:52.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561013aa00 0x7f56101369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:52.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56100ab5f0 0x7f561012e210 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:52.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 --2- 172.21.15.151:0/2975752585 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f56100ab220 0x7f56100b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:52.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 >> 172.21.15.151:0/2975752585 conn(0x7f5610070510 msgr2=0x7f5610132410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:52.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 shutdown_connections 2023-09-30T02:01:52.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:52.147+0000 7f560f59e700 1 -- 172.21.15.151:0/2975752585 wait complete. 2023-09-30T02:01:55.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:55.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 Processor -- start 2023-09-30T02:01:55.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- start start 2023-09-30T02:01:55.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 0x7f01f80ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:55.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f806c8e0 con 0x7f01f806b550 2023-09-30T02:01:55.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80abe70 con 0x7f01f80ab460 2023-09-30T02:01:55.276 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f8141ce0 con 0x7f01f806b180 2023-09-30T02:01:55.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f6d9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f01f806b550 0x7f01f813e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58402/0 (socket says 172.21.15.151:58402) 2023-09-30T02:01:55.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f6d9d700 1 -- 172.21.15.151:0/2386152496 learned_addr learned my addr 172.21.15.151:0/2386152496 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:01:55.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f659c700 1 --2- 172.21.15.151:0/2386152496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 0x7f01f80ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:55.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f659c700 1 -- 172.21.15.151:0/2386152496 >> v1:172.21.15.172:6789/0 conn(0x7f01f80ab460 legacy=0x7f01f81415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f659c700 1 -- 172.21.15.151:0/2386152496 >> v1:172.21.15.151:6789/0 conn(0x7f01f806b550 legacy=0x7f01f813e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f659c700 1 -- 172.21.15.151:0/2386152496 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80a9b00 con 0x7f01f806b180 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f659c700 1 --2- 172.21.15.151:0/2386152496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 0x7f01f80ac390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009910 tx=0x7f01e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76140eee65721dce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f5d9b700 1 -- 172.21.15.151:0/2386152496 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800b860 con 0x7f01f806b180 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f5d9b700 1 -- 172.21.15.151:0/2386152496 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e800b9c0 con 0x7f01f806b180 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f5d9b700 1 -- 172.21.15.151:0/2386152496 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800bd60 con 0x7f01f806b180 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- 172.21.15.151:0/2386152496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 msgr2=0x7f01f80ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 --2- 172.21.15.151:0/2386152496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 0x7f01f80ac390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009910 tx=0x7f01e80091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- 172.21.15.151:0/2386152496 shutdown_connections 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 --2- 172.21.15.151:0/2386152496 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f806b180 0x7f01f80ac390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.271+0000 7f01f759e700 1 -- 172.21.15.151:0/2386152496 >> 172.21.15.151:0/2386152496 conn(0x7f01f8070510 msgr2=0x7f01f8070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:55.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- 172.21.15.151:0/2386152496 shutdown_connections 2023-09-30T02:01:55.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- 172.21.15.151:0/2386152496 wait complete. 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 Processor -- start 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- start start 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 0x7f01f80afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01f806b550 0x7f01f80adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 0x7f01f81547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80af8a0 con 0x7f01f806b180 2023-09-30T02:01:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80ad720 con 0x7f01f806b550 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f8144b80 con 0x7f01f80ab460 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 0x7f01f80afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 0x7f01f81547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f659c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 0x7f01f80afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40788/0 (socket says 172.21.15.151:40788) 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f659c700 1 -- 172.21.15.151:0/456303557 learned_addr learned my addr 172.21.15.151:0/456303557 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:55.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01edd9b700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01f806b550 0x7f01f80adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:55.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 -- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01f806b550 msgr2=0x7f01f80adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01f806b550 0x7f01f80adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 -- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 msgr2=0x7f01f80afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 0x7f01f80afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 -- 172.21.15.151:0/456303557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8009420 con 0x7f01f80ab460 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f6d9d700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 0x7f01f81547b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01f000cb90 tx=0x7f01f00019a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000a8b0 con 0x7f01f80ab460 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f000aa10 con 0x7f01f80ab460 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000add0 con 0x7f01f80ab460 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f81457c0 con 0x7f01f80ab460 2023-09-30T02:01:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.275+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80ae8c0 con 0x7f01f80ab460 2023-09-30T02:01:55.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f01f0007300 con 0x7f01f80ab460 2023-09-30T02:01:55.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f80008d0 con 0x7f01f80ab460 2023-09-30T02:01:55.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01eeffd700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 0x7f01d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:55.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f01f007c870 con 0x7f01f80ab460 2023-09-30T02:01:55.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01f659c700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 0x7f01d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:55.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.279+0000 7f01f659c700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 0x7f01d8062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009fd0 tx=0x7f01e8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:55.292 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.287+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f01f004d4a0 con 0x7f01f80ab460 2023-09-30T02:01:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.483+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01f80ad8e0 con 0x7f01f80ab460 2023-09-30T02:01:55.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.483+0000 7f01eeffd700 1 -- 172.21.15.151:0/456303557 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f01f00506f0 con 0x7f01f80ab460 2023-09-30T02:01:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:55.488 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.483+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 msgr2=0x7f01d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.483+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 0x7f01d8062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009fd0 tx=0x7f01e8017000 comp rx=0 tx=0).stop 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 msgr2=0x7f01f81547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 0x7f01f81547b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01f000cb90 tx=0x7f01f00019a0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 shutdown_connections 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f01d8060550 0x7f01d8062a00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01f80ab460 0x7f01f81547b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f01f806b550 0x7f01f80adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 --2- 172.21.15.151:0/456303557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01f806b180 0x7f01f80afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:55.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 >> 172.21.15.151:0/456303557 conn(0x7f01f8070510 msgr2=0x7f01f813b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:55.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 shutdown_connections 2023-09-30T02:01:55.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:55.487+0000 7f01f759e700 1 -- 172.21.15.151:0/456303557 wait complete. 2023-09-30T02:01:58.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:01:58.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 Processor -- start 2023-09-30T02:01:58.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 -- start start 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65d80ac730 0x7f65d80a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d813ad00 con 0x7f65d8070740 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d80a5090 con 0x7f65d80ac730 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65df25a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d8070b10 con 0x7f65d813a6a0 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65dd7f7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65dd7f7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43240/0 (socket says 172.21.15.151:43240) 2023-09-30T02:01:58.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65dd7f7700 1 -- 172.21.15.151:0/2033534822 learned_addr learned my addr 172.21.15.151:0/2033534822 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:58.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.611+0000 7f65d7fff700 1 --2- 172.21.15.151:0/2033534822 >> v2:172.21.15.172:3300/0 conn(0x7f65d80ac730 0x7f65d80a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65dd7f7700 1 -- 172.21.15.151:0/2033534822 >> v2:172.21.15.172:3300/0 conn(0x7f65d80ac730 msgr2=0x7f65d80a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:01:58.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65dd7f7700 1 --2- 172.21.15.151:0/2033534822 >> v2:172.21.15.172:3300/0 conn(0x7f65d80ac730 0x7f65d80a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65dd7f7700 1 -- 172.21.15.151:0/2033534822 >> v1:172.21.15.172:6789/0 conn(0x7f65d8070740 legacy=0x7f65d80a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65dd7f7700 1 -- 172.21.15.151:0/2033534822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d80a6fc0 con 0x7f65d813a6a0 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65dd7f7700 1 --2- 172.21.15.151:0/2033534822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009f20 tx=0x7f65d0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=182b6c813513fc61 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65d77fe700 1 -- 172.21.15.151:0/2033534822 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000e070 con 0x7f65d813a6a0 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65d77fe700 1 -- 172.21.15.151:0/2033534822 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000b9d0 con 0x7f65d813a6a0 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65d77fe700 1 -- 172.21.15.151:0/2033534822 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d00092e0 con 0x7f65d813a6a0 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- 172.21.15.151:0/2033534822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 msgr2=0x7f65d80a98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 --2- 172.21.15.151:0/2033534822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009f20 tx=0x7f65d0009840 comp rx=0 tx=0).stop 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- 172.21.15.151:0/2033534822 shutdown_connections 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 --2- 172.21.15.151:0/2033534822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d813a6a0 0x7f65d80a98d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 --2- 172.21.15.151:0/2033534822 >> v2:172.21.15.172:3300/0 conn(0x7f65d80ac730 0x7f65d80a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- 172.21.15.151:0/2033534822 >> 172.21.15.151:0/2033534822 conn(0x7f65d8067710 msgr2=0x7f65d8067b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:58.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- 172.21.15.151:0/2033534822 shutdown_connections 2023-09-30T02:01:58.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- 172.21.15.151:0/2033534822 wait complete. 2023-09-30T02:01:58.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 Processor -- start 2023-09-30T02:01:58.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.615+0000 7f65df25a700 1 -- start start 2023-09-30T02:01:58.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d8070740 0x7f65d806b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 0x7f65d813c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d813d8a0 con 0x7f65d80ac730 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d813cde0 con 0x7f65d813a6a0 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d813daa0 con 0x7f65d8070740 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:40804/0 (socket says 172.21.15.151:40804) 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 -- 172.21.15.151:0/3948143564 learned_addr learned my addr 172.21.15.151:0/3948143564 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65dcff6700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d8070740 0x7f65d806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65dd7f7700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 0x7f65d813c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 -- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d8070740 msgr2=0x7f65d806b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d8070740 0x7f65d806b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 -- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 msgr2=0x7f65d813c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 0x7f65d813c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 -- 172.21.15.151:0/3948143564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0009ae0 con 0x7f65d80ac730 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65dd7f7700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 0x7f65d813c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d7fff700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ec80 tx=0x7f65c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c400c850 con 0x7f65d80ac730 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c400c9d0 con 0x7f65d80ac730 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d813dd80 con 0x7f65d80ac730 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c400cd90 con 0x7f65d80ac730 2023-09-30T02:01:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d81395a0 con 0x7f65d80ac730 2023-09-30T02:01:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d80670a0 con 0x7f65d80ac730 2023-09-30T02:01:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.619+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f65c4013680 con 0x7f65d80ac730 2023-09-30T02:01:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.623+0000 7f65d4ff9700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 0x7f65bc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:01:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.623+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f65c407d150 con 0x7f65d80ac730 2023-09-30T02:01:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.623+0000 7f65dcff6700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 0x7f65bc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:01:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.623+0000 7f65dcff6700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 0x7f65bc062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f65cc003a80 tx=0x7f65cc007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:01:58.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.627+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f65c404d930 con 0x7f65d80ac730 2023-09-30T02:01:58.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.819+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65d813cf20 con 0x7f65d80ac730 2023-09-30T02:01:58.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.819+0000 7f65d4ff9700 1 -- 172.21.15.151:0/3948143564 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f65c4050f50 con 0x7f65d80ac730 2023-09-30T02:01:58.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:01:58.827 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:01:58.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 msgr2=0x7f65bc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 0x7f65bc062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f65cc003a80 tx=0x7f65cc007ec0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 msgr2=0x7f65d8139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:01:58.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ec80 tx=0x7f65c40091a0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 shutdown_connections 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f65bc060550 0x7f65bc062a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65d813a6a0 0x7f65d813c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d80ac730 0x7f65d8139060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 --2- 172.21.15.151:0/3948143564 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d8070740 0x7f65d806b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:01:58.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 >> 172.21.15.151:0/3948143564 conn(0x7f65d8067710 msgr2=0x7f65d80ab4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:01:58.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 shutdown_connections 2023-09-30T02:01:58.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:01:58.823+0000 7f65df25a700 1 -- 172.21.15.151:0/3948143564 wait complete. 2023-09-30T02:02:01.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:01.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.947+0000 7f263f59e700 1 Processor -- start 2023-09-30T02:02:01.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.947+0000 7f263f59e700 1 -- start start 2023-09-30T02:02:01.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 0x7f26400ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f264006b550 0x7f264013e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264006c8e0 con 0x7f26400ab460 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2640141ce0 con 0x7f264006b180 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26400abe70 con 0x7f264006b550 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263e59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 0x7f26400ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263dd9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f26400ab460 0x7f26401415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38722/0 (socket says 172.21.15.151:38722) 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263dd9b700 1 -- 172.21.15.151:0/2430809639 learned_addr learned my addr 172.21.15.151:0/2430809639 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 --2- 172.21.15.151:0/2430809639 >> v2:172.21.15.172:3300/0 conn(0x7f264006b550 0x7f264013e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789530055 0 0) 0x7f264006c8e0 con 0x7f26400ab460 2023-09-30T02:02:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628003570 con 0x7f26400ab460 2023-09-30T02:02:01.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 -- 172.21.15.151:0/2430809639 >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 msgr2=0x7f26400ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257719546 0 0) 0x7f2628003570 con 0x7f26400ab460 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 --2- 172.21.15.151:0/2430809639 >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 0x7f26400ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 -- 172.21.15.151:0/2430809639 >> v1:172.21.15.151:6789/0 conn(0x7f26400ab460 legacy=0x7f26401415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 -- 172.21.15.151:0/2430809639 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400a9b00 con 0x7f264006b550 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263e59c700 1 --2- 172.21.15.151:0/2430809639 >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 0x7f26400ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:01.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263ed9d700 1 --2- 172.21.15.151:0/2430809639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264006b550 0x7f264013e070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f264013f2c0 tx=0x7f26380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834f637f94df5397 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300017f0 con 0x7f264006b550 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2638011420 con 0x7f264006b550 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263d59a700 1 -- 172.21.15.151:0/2430809639 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380117c0 con 0x7f264006b550 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- 172.21.15.151:0/2430809639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264006b550 msgr2=0x7f264013e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 --2- 172.21.15.151:0/2430809639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264006b550 0x7f264013e070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f264013f2c0 tx=0x7f26380091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- 172.21.15.151:0/2430809639 shutdown_connections 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 --2- 172.21.15.151:0/2430809639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f264006b550 0x7f264013e070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 --2- 172.21.15.151:0/2430809639 >> v2:172.21.15.151:3300/0 conn(0x7f264006b180 0x7f26400ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- 172.21.15.151:0/2430809639 >> 172.21.15.151:0/2430809639 conn(0x7f2640070510 msgr2=0x7f2640070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- 172.21.15.151:0/2430809639 shutdown_connections 2023-09-30T02:02:01.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.951+0000 7f263f59e700 1 -- 172.21.15.151:0/2430809639 wait complete. 2023-09-30T02:02:01.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 Processor -- start 2023-09-30T02:02:01.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- start start 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 0x7f26400af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26400ab460 0x7f2640154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640145020 con 0x7f264006b180 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26400a4b80 con 0x7f26400ab460 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400a4df0 con 0x7f264006b550 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263e59c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 0x7f26400af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53800/0 (socket says 172.21.15.151:53800) 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 -- 172.21.15.151:0/3063481702 learned_addr learned my addr 172.21.15.151:0/3063481702 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:01.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263ed9d700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26400ab460 0x7f2640154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 -- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26400ab460 msgr2=0x7f2640154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26400ab460 0x7f2640154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 -- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 msgr2=0x7f26400af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 0x7f26400af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 -- 172.21.15.151:0/3063481702 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634009420 con 0x7f264006b550 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263e59c700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 0x7f26400af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263dd9b700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f263000ccf0 tx=0x7f26300016f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630004dc0 con 0x7f264006b550 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263000a620 con 0x7f264006b550 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2638009420 con 0x7f264006b550 2023-09-30T02:02:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263000a9e0 con 0x7f264006b550 2023-09-30T02:02:01.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.955+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2640144d20 con 0x7f264006b550 2023-09-30T02:02:01.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.959+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26401451d0 con 0x7f264006b550 2023-09-30T02:02:01.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.959+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f263000ad50 con 0x7f264006b550 2023-09-30T02:02:01.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.959+0000 7f262e7fc700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 0x7f2620062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:01.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.959+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f263007c050 con 0x7f264006b550 2023-09-30T02:02:01.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.959+0000 7f263e59c700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 0x7f2620062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:01.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.963+0000 7f263e59c700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 0x7f2620062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2634011600 tx=0x7f26340091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:01.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:01.967+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f26401451d0 con 0x7f264006b550 2023-09-30T02:02:02.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.163+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f26401451d0 con 0x7f264006b550 2023-09-30T02:02:02.168 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.163+0000 7f262e7fc700 1 -- 172.21.15.151:0/3063481702 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f26401451d0 con 0x7f264006b550 2023-09-30T02:02:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:02.169 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:02.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 msgr2=0x7f2620062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:02.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 0x7f2620062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2634011600 tx=0x7f26340091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 msgr2=0x7f2640142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:02.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f263000ccf0 tx=0x7f26300016f0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 shutdown_connections 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2620060550 0x7f2620062a00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26400ab460 0x7f2640154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f264006b550 0x7f2640142770 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f263000ccf0 tx=0x7f26300016f0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 --2- 172.21.15.151:0/3063481702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f264006b180 0x7f26400af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 >> 172.21.15.151:0/3063481702 conn(0x7f2640070510 msgr2=0x7f26400aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 shutdown_connections 2023-09-30T02:02:02.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:02.167+0000 7f263f59e700 1 -- 172.21.15.151:0/3063481702 wait complete. 2023-09-30T02:02:05.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:05.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 Processor -- start 2023-09-30T02:02:05.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 -- start start 2023-09-30T02:02:05.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 0x7f7518137f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 0x7f7518138490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180acd90 con 0x7f751813e8d0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751806e840 con 0x7f751806e470 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7517fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180ad000 con 0x7f7518067fa0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75177fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f751813e8d0 0x7f751813ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38728/0 (socket says 172.21.15.151:38728) 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75167fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 0x7f7518138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75177fe700 1 -- 172.21.15.151:0/3269750136 learned_addr learned my addr 172.21.15.151:0/3269750136 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7516ffd700 1 --2- 172.21.15.151:0/3269750136 >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 0x7f7518137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7515ffb700 1 -- 172.21.15.151:0/3269750136 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847577432 0 0) 0x7f75180acd90 con 0x7f751813e8d0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f7515ffb700 1 -- 172.21.15.151:0/3269750136 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8003570 con 0x7f751813e8d0 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75167fc700 1 -- 172.21.15.151:0/3269750136 >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 msgr2=0x7f7518137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75167fc700 1 --2- 172.21.15.151:0/3269750136 >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 0x7f7518137f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.299+0000 7f75167fc700 1 -- 172.21.15.151:0/3269750136 >> v1:172.21.15.151:6789/0 conn(0x7f751813e8d0 legacy=0x7f751813ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f75167fc700 1 -- 172.21.15.151:0/3269750136 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751813f440 con 0x7f7518067fa0 2023-09-30T02:02:05.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7516ffd700 1 --2- 172.21.15.151:0/3269750136 >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 0x7f7518137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:05.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f75167fc700 1 --2- 172.21.15.151:0/3269750136 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 0x7f7518138490 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f75000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83100110132dd09b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:05.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7515ffb700 1 -- 172.21.15.151:0/3269750136 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000b640 con 0x7f7518067fa0 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7515ffb700 1 -- 172.21.15.151:0/3269750136 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000b7a0 con 0x7f7518067fa0 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7515ffb700 1 -- 172.21.15.151:0/3269750136 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000bb40 con 0x7f7518067fa0 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 -- 172.21.15.151:0/3269750136 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 msgr2=0x7f7518138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 --2- 172.21.15.151:0/3269750136 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 0x7f7518138490 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f75000091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 -- 172.21.15.151:0/3269750136 shutdown_connections 2023-09-30T02:02:05.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 --2- 172.21.15.151:0/3269750136 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7518067fa0 0x7f7518138490 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 --2- 172.21.15.151:0/3269750136 >> v2:172.21.15.172:3300/0 conn(0x7f751806e470 0x7f7518137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 -- 172.21.15.151:0/3269750136 >> 172.21.15.151:0/3269750136 conn(0x7f7518070510 msgr2=0x7f7518070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 -- 172.21.15.151:0/3269750136 shutdown_connections 2023-09-30T02:02:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.303+0000 7f7517fff700 1 -- 172.21.15.151:0/3269750136 wait complete. 2023-09-30T02:02:05.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 Processor -- start 2023-09-30T02:02:05.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 -- start start 2023-09-30T02:02:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7518067fa0 0x7f75180a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f751813e8d0 0x7f75180aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518134890 con 0x7f751806e470 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75180b1750 con 0x7f7518067fa0 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7517fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180b12c0 con 0x7f751813e8d0 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39684/0 (socket says 172.21.15.151:39684) 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 -- 172.21.15.151:0/3169474949 learned_addr learned my addr 172.21.15.151:0/3169474949 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f7516ffd700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7518067fa0 0x7f75180a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75177fe700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f751813e8d0 0x7f75180aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 -- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f751813e8d0 msgr2=0x7f75180aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f751813e8d0 0x7f75180aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 -- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7518067fa0 msgr2=0x7f75180a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7518067fa0 0x7f75180a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 -- 172.21.15.151:0/3169474949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009420 con 0x7f751806e470 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.307+0000 7f75167fc700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f75000093f0 tx=0x7f7500017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500006af0 con 0x7f751806e470 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750001b960 con 0x7f751806e470 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75180aab00 con 0x7f751806e470 2023-09-30T02:02:05.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750001bd20 con 0x7f751806e470 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75180a55e0 con 0x7f751806e470 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75180a57d0 con 0x7f751806e470 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7500023070 con 0x7f751806e470 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 0x7f74fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f750007d940 con 0x7f751806e470 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f7516ffd700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 0x7f74fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:05.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.311+0000 7f7516ffd700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 0x7f74fc062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7508006f00 tx=0x7f7508008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:05.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.323+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f750004e080 con 0x7f751806e470 2023-09-30T02:02:05.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.511+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f75180008d0 con 0x7f751806e470 2023-09-30T02:02:05.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.511+0000 7f750effd700 1 -- 172.21.15.151:0/3169474949 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f75000516a0 con 0x7f751806e470 2023-09-30T02:02:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:05.518 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:05.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 msgr2=0x7f74fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 0x7f74fc062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7508006f00 tx=0x7f7508008040 comp rx=0 tx=0).stop 2023-09-30T02:02:05.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 msgr2=0x7f75180aa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f75000093f0 tx=0x7f7500017000 comp rx=0 tx=0).stop 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 shutdown_connections 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f74fc060550 0x7f74fc062a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f751813e8d0 0x7f75180aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f751806e470 0x7f75180aa080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 --2- 172.21.15.151:0/3169474949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7518067fa0 0x7f75180a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 >> 172.21.15.151:0/3169474949 conn(0x7f7518070510 msgr2=0x7f7518135ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 shutdown_connections 2023-09-30T02:02:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:05.515+0000 7f7517fff700 1 -- 172.21.15.151:0/3169474949 wait complete. 2023-09-30T02:02:08.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:08.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 Processor -- start 2023-09-30T02:02:08.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 -- start start 2023-09-30T02:02:08.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f94f00b0070 0x7f94f00aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f0138dc0 con 0x7f94f00aaec0 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f00abcb0 con 0x7f94f00b0070 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94f4c87700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00abeb0 con 0x7f94f00b0440 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53830/0 (socket says 172.21.15.151:53830) 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 -- 172.21.15.151:0/3777769789 learned_addr learned my addr 172.21.15.151:0/3777769789 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94ee59c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f94f00b0070 0x7f94f00aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 -- 172.21.15.151:0/3777769789 >> v2:172.21.15.151:3300/0 conn(0x7f94f00b0070 msgr2=0x7f94f00aa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 --2- 172.21.15.151:0/3777769789 >> v2:172.21.15.151:3300/0 conn(0x7f94f00b0070 0x7f94f00aa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 -- 172.21.15.151:0/3777769789 >> v1:172.21.15.172:6789/0 conn(0x7f94f00aaec0 legacy=0x7f94f00a8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 -- 172.21.15.151:0/3777769789 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f00ac190 con 0x7f94f00b0440 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94eed9d700 1 --2- 172.21.15.151:0/3777769789 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f94e80099b0 tx=0x7f94e8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f0205423f5b0fa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.639+0000 7f94ed59a700 1 -- 172.21.15.151:0/3777769789 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8009ea0 con 0x7f94f00b0440 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94ed59a700 1 -- 172.21.15.151:0/3777769789 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e800b860 con 0x7f94f00b0440 2023-09-30T02:02:08.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94ed59a700 1 -- 172.21.15.151:0/3777769789 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e800bc20 con 0x7f94f00b0440 2023-09-30T02:02:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- 172.21.15.151:0/3777769789 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 msgr2=0x7f94f013b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- 172.21.15.151:0/3777769789 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f94e80099b0 tx=0x7f94e8009250 comp rx=0 tx=0).stop 2023-09-30T02:02:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- 172.21.15.151:0/3777769789 shutdown_connections 2023-09-30T02:02:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- 172.21.15.151:0/3777769789 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f013b1a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- 172.21.15.151:0/3777769789 >> v2:172.21.15.151:3300/0 conn(0x7f94f00b0070 0x7f94f00aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- 172.21.15.151:0/3777769789 >> 172.21.15.151:0/3777769789 conn(0x7f94f0070fe0 msgr2=0x7f94f00af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- 172.21.15.151:0/3777769789 shutdown_connections 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- 172.21.15.151:0/3777769789 wait complete. 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 Processor -- start 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- start start 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f00aaec0 0x7f94f00a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f00a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00a86c0 con 0x7f94f00b0070 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00a89a0 con 0x7f94f00aaec0 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94f4c87700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00a6810 con 0x7f94f00b0440 2023-09-30T02:02:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94edd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94edd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39710/0 (socket says 172.21.15.151:39710) 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94eed9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f00a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94edd9b700 1 -- 172.21.15.151:0/1135468728 learned_addr learned my addr 172.21.15.151:0/1135468728 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.643+0000 7f94ee59c700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f00aaec0 0x7f94f00a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 -- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 msgr2=0x7f94f00a62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f00a62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 -- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f00aaec0 msgr2=0x7f94f00a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f00aaec0 0x7f94f00a4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 -- 172.21.15.151:0/1135468728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80094f0 con 0x7f94f00b0070 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94eed9d700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f00a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:08.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94edd9b700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94e000cb40 tx=0x7f94e0001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0002060 con 0x7f94f00b0070 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000a780 con 0x7f94f00b0070 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f00a7070 con 0x7f94f00b0070 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e000ab40 con 0x7f94f00b0070 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f00a7580 con 0x7f94f00b0070 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.647+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f00a7800 con 0x7f94f00b0070 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.651+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f94e0003030 con 0x7f94f00b0070 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.651+0000 7f94de7fc700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 0x7f94d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.651+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f94e007c490 con 0x7f94f00b0070 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.651+0000 7f94ee59c700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 0x7f94d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.651+0000 7f94ee59c700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 0x7f94d0062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009db0 tx=0x7f94e40091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:08.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.659+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f94f00a7800 con 0x7f94f00b0070 2023-09-30T02:02:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94f00a7800 con 0x7f94f00b0070 2023-09-30T02:02:08.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94de7fc700 1 -- 172.21.15.151:0/1135468728 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f94f00a7800 con 0x7f94f00b0070 2023-09-30T02:02:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:08.855 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 msgr2=0x7f94d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 0x7f94d0062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009db0 tx=0x7f94e40091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 msgr2=0x7f94f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94e000cb40 tx=0x7f94e0001770 comp rx=0 tx=0).stop 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 shutdown_connections 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f94d0060550 0x7f94d0062a00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94f00b0440 0x7f94f00a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94f00b0070 0x7f94f006d450 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 --2- 172.21.15.151:0/1135468728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94f00aaec0 0x7f94f00a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:08.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 >> 172.21.15.151:0/1135468728 conn(0x7f94f0070fe0 msgr2=0x7f94f0139e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:08.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 shutdown_connections 2023-09-30T02:02:08.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:08.851+0000 7f94f4c87700 1 -- 172.21.15.151:0/1135468728 wait complete. 2023-09-30T02:02:11.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:11.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 Processor -- start 2023-09-30T02:02:11.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- start start 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f269c06b550 0x7f269c13e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c06c8e0 con 0x7f269c06b180 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269c141ce0 con 0x7f269c06b550 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0abe70 con 0x7f269c0ab460 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a0b1d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f269c06b180 0x7f269c1415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60144/0 (socket says 172.21.15.151:60144) 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a0b1d700 1 -- 172.21.15.151:0/4144421632 learned_addr learned my addr 172.21.15.151:0/4144421632 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:11.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a131e700 1 --2- 172.21.15.151:0/4144421632 >> v2:172.21.15.151:3300/0 conn(0x7f269c06b550 0x7f269c13e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269b7fe700 1 -- 172.21.15.151:0/4144421632 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729479560 0 0) 0x7f269c06c8e0 con 0x7f269c06b180 2023-09-30T02:02:11.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269b7fe700 1 -- 172.21.15.151:0/4144421632 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003570 con 0x7f269c06b180 2023-09-30T02:02:11.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 -- 172.21.15.151:0/4144421632 >> v2:172.21.15.151:3300/0 conn(0x7f269c06b550 msgr2=0x7f269c13e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:11.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 --2- 172.21.15.151:0/4144421632 >> v2:172.21.15.151:3300/0 conn(0x7f269c06b550 0x7f269c13e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:11.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 -- 172.21.15.151:0/4144421632 >> v1:172.21.15.151:6789/0 conn(0x7f269c06b180 legacy=0x7f269c1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:11.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 -- 172.21.15.151:0/4144421632 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c0a9b00 con 0x7f269c0ab460 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269bfff700 1 --2- 172.21.15.151:0/4144421632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ac390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f268c009e60 tx=0x7f268c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc85714f479244a4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269b7fe700 1 -- 172.21.15.151:0/4144421632 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00e070 con 0x7f269c0ab460 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269b7fe700 1 -- 172.21.15.151:0/4144421632 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c00bab0 con 0x7f269c0ab460 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f269b7fe700 1 -- 172.21.15.151:0/4144421632 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00be50 con 0x7f269c0ab460 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- 172.21.15.151:0/4144421632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 msgr2=0x7f269c0ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/4144421632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ac390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f268c009e60 tx=0x7f268c009780 comp rx=0 tx=0).stop 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- 172.21.15.151:0/4144421632 shutdown_connections 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/4144421632 >> v2:172.21.15.151:3300/0 conn(0x7f269c06b550 0x7f269c13e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/4144421632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ac390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- 172.21.15.151:0/4144421632 >> 172.21.15.151:0/4144421632 conn(0x7f269c070510 msgr2=0x7f269c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- 172.21.15.151:0/4144421632 shutdown_connections 2023-09-30T02:02:11.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.979+0000 7f26a1b1f700 1 -- 172.21.15.151:0/4144421632 wait complete. 2023-09-30T02:02:11.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 Processor -- start 2023-09-30T02:02:11.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- start start 2023-09-30T02:02:11.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 0x7f269c140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c135f00 con 0x7f269c06b550 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ad6b0 con 0x7f269c06b180 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ad920 con 0x7f269c0ab460 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42270/0 (socket says 172.21.15.151:42270) 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 -- 172.21.15.151:0/1056908920 learned_addr learned my addr 172.21.15.151:0/1056908920 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a131e700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a0b1d700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 0x7f269c140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 -- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 msgr2=0x7f269c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 -- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 msgr2=0x7f269c140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:11.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 0x7f269c140a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 -- 172.21.15.151:0/1056908920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c009a20 con 0x7f269c06b550 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a0b1d700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 0x7f269c140a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a131e700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f269bfff700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b610 tx=0x7f268c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c012af0 con 0x7f269c06b550 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c012c70 con 0x7f269c06b550 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c0ae7c0 con 0x7f269c06b550 2023-09-30T02:02:11.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c0263f0 con 0x7f269c06b550 2023-09-30T02:02:11.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.983+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c0b04b0 con 0x7f269c06b550 2023-09-30T02:02:11.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.987+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c0008d0 con 0x7f269c06b550 2023-09-30T02:02:11.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.987+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f268c026840 con 0x7f269c06b550 2023-09-30T02:02:11.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.987+0000 7f2698ff9700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 0x7f267c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:11.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.987+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f268c07e340 con 0x7f269c06b550 2023-09-30T02:02:11.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.987+0000 7f26a0b1d700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 0x7f267c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:11.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.991+0000 7f26a0b1d700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 0x7f267c062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2684007fd0 tx=0x7f2684009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:11.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:11.995+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f268c04eb20 con 0x7f269c06b550 2023-09-30T02:02:12.197 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.191+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f269c06d0b0 con 0x7f269c06b550 2023-09-30T02:02:12.198 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.191+0000 7f2698ff9700 1 -- 172.21.15.151:0/1056908920 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f268c052140 con 0x7f269c06b550 2023-09-30T02:02:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:12.198 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 msgr2=0x7f267c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 0x7f267c062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2684007fd0 tx=0x7f2684009040 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 msgr2=0x7f269c0add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b610 tx=0x7f268c019000 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 shutdown_connections 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f267c060550 0x7f267c062a00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f269c0ab460 0x7f269c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f269c06b550 0x7f269c0add40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 --2- 172.21.15.151:0/1056908920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c06b180 0x7f269c140a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:12.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 >> 172.21.15.151:0/1056908920 conn(0x7f269c070510 msgr2=0x7f269c0aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:12.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 shutdown_connections 2023-09-30T02:02:12.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:12.195+0000 7f26a1b1f700 1 -- 172.21.15.151:0/1056908920 wait complete. 2023-09-30T02:02:15.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:15.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 Processor -- start 2023-09-30T02:02:15.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 -- start start 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 0x7f6b0c13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c0ab3e0 con 0x7f6b0c0aae90 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c070b10 con 0x7f6b0c13a050 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b10ad0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0ab630 con 0x7f6b0c070740 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0b7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 0x7f6b0c13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0affd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6b0c0aae90 0x7f6b0c13dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60156/0 (socket says 172.21.15.151:60156) 2023-09-30T02:02:15.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0affd700 1 -- 172.21.15.151:0/3502510256 learned_addr learned my addr 172.21.15.151:0/3502510256 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:15.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0b7fe700 1 -- 172.21.15.151:0/3502510256 >> v1:172.21.15.172:6789/0 conn(0x7f6b0c13a050 legacy=0x7f6b0c13a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0b7fe700 1 -- 172.21.15.151:0/3502510256 >> v1:172.21.15.151:6789/0 conn(0x7f6b0c0aae90 legacy=0x7f6b0c13dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0a7fc700 1 -- 172.21.15.151:0/3502510256 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464234776 0 0) 0x7f6b0c0ab3e0 con 0x7f6b0c0aae90 2023-09-30T02:02:15.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0b7fe700 1 -- 172.21.15.151:0/3502510256 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0a8850 con 0x7f6b0c070740 2023-09-30T02:02:15.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0b7fe700 1 --2- 172.21.15.151:0/3502510256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 0x7f6b0c13d6c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009910 tx=0x7f6afc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7de01fbc13350a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:15.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.319+0000 7f6b0a7fc700 1 -- 172.21.15.151:0/3502510256 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00b860 con 0x7f6b0c070740 2023-09-30T02:02:15.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b0a7fc700 1 -- 172.21.15.151:0/3502510256 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc00b9c0 con 0x7f6b0c070740 2023-09-30T02:02:15.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b0a7fc700 1 -- 172.21.15.151:0/3502510256 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00bd60 con 0x7f6b0c070740 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- 172.21.15.151:0/3502510256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 msgr2=0x7f6b0c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/3502510256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 0x7f6b0c13d6c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009910 tx=0x7f6afc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- 172.21.15.151:0/3502510256 shutdown_connections 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/3502510256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c070740 0x7f6b0c13d6c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- 172.21.15.151:0/3502510256 >> 172.21.15.151:0/3502510256 conn(0x7f6b0c0b03f0 msgr2=0x7f6b0c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- 172.21.15.151:0/3502510256 shutdown_connections 2023-09-30T02:02:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- 172.21.15.151:0/3502510256 wait complete. 2023-09-30T02:02:15.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 Processor -- start 2023-09-30T02:02:15.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.323+0000 7f6b10ad0700 1 -- start start 2023-09-30T02:02:15.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 0x7f6b0c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:15.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c13a050 0x7f6b0c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c06dc00 con 0x7f6b0c0aae90 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c06eb00 con 0x7f6b0c070740 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c06e7b0 con 0x7f6b0c13a050 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c13a050 0x7f6b0c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:15.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:42282/0 (socket says 172.21.15.151:42282) 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 -- 172.21.15.151:0/2597322094 learned_addr learned my addr 172.21.15.151:0/2597322094 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0b7fe700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 0x7f6b0c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 -- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c13a050 msgr2=0x7f6b0c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c13a050 0x7f6b0c06bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 -- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 msgr2=0x7f6b0c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 0x7f6b0c06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 -- 172.21.15.151:0/2597322094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc009420 con 0x7f6b0c0aae90 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0b7fe700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 0x7f6b0c06e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b0affd700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6af400c810 tx=0x7f6af4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4005c00 con 0x7f6b0c0aae90 2023-09-30T02:02:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4005d60 con 0x7f6b0c0aae90 2023-09-30T02:02:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c06c920 con 0x7f6b0c0aae90 2023-09-30T02:02:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4007320 con 0x7f6b0c0aae90 2023-09-30T02:02:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c141550 con 0x7f6b0c0aae90 2023-09-30T02:02:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c0008d0 con 0x7f6b0c0aae90 2023-09-30T02:02:15.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.327+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6af401b3b0 con 0x7f6b0c0aae90 2023-09-30T02:02:15.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.331+0000 7f6b037fe700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 0x7f6aec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:15.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.331+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6af407beb0 con 0x7f6b0c0aae90 2023-09-30T02:02:15.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.331+0000 7f6b0b7fe700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 0x7f6aec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:15.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.331+0000 7f6b0b7fe700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 0x7f6aec062a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0093f0 tx=0x7f6afc017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:15.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.335+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6af404c710 con 0x7f6b0c0aae90 2023-09-30T02:02:15.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.535+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b0c06f140 con 0x7f6b0c0aae90 2023-09-30T02:02:15.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.535+0000 7f6b037fe700 1 -- 172.21.15.151:0/2597322094 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6af4017080 con 0x7f6b0c0aae90 2023-09-30T02:02:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:15.542 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.539+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 msgr2=0x7f6aec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.539+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 0x7f6aec062a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0093f0 tx=0x7f6afc017000 comp rx=0 tx=0).stop 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.539+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 msgr2=0x7f6b0c06ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.539+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6af400c810 tx=0x7f6af4007d90 comp rx=0 tx=0).stop 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 shutdown_connections 2023-09-30T02:02:15.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6aec060550 0x7f6aec062a00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6b0c13a050 0x7f6b0c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6b0c0aae90 0x7f6b0c06ba90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 --2- 172.21.15.151:0/2597322094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b0c070740 0x7f6b0c06e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 >> 172.21.15.151:0/2597322094 conn(0x7f6b0c0b03f0 msgr2=0x7f6b0c13b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 shutdown_connections 2023-09-30T02:02:15.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:15.543+0000 7f6b10ad0700 1 -- 172.21.15.151:0/2597322094 wait complete. 2023-09-30T02:02:18.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:18.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 Processor -- start 2023-09-30T02:02:18.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- start start 2023-09-30T02:02:18.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30a4139f20 0x7f30a40aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a406c3f0 con 0x7f30a40b0a80 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a406db10 con 0x7f30a4139f20 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40b0e50 con 0x7f30a40aa9e0 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49266/0 (socket says 172.21.15.151:49266) 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 -- 172.21.15.151:0/688157424 learned_addr learned my addr 172.21.15.151:0/688157424 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:18.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a2ffd700 1 --2- 172.21.15.151:0/688157424 >> v2:172.21.15.172:3300/0 conn(0x7f30a4139f20 0x7f30a40aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 -- 172.21.15.151:0/688157424 >> v2:172.21.15.172:3300/0 conn(0x7f30a4139f20 msgr2=0x7f30a40aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:18.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 --2- 172.21.15.151:0/688157424 >> v2:172.21.15.172:3300/0 conn(0x7f30a4139f20 0x7f30a40aa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 -- 172.21.15.151:0/688157424 >> v1:172.21.15.172:6789/0 conn(0x7f30a40b0a80 legacy=0x7f30a406e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 -- 172.21.15.151:0/688157424 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a406dd80 con 0x7f30a40aa9e0 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a3fff700 1 --2- 172.21.15.151:0/688157424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f309c0099b0 tx=0x7f309c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95897f72964cf1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a27fc700 1 -- 172.21.15.151:0/688157424 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c009ea0 con 0x7f30a40aa9e0 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a27fc700 1 -- 172.21.15.151:0/688157424 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00b860 con 0x7f30a40aa9e0 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a27fc700 1 -- 172.21.15.151:0/688157424 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00bc00 con 0x7f30a40aa9e0 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- 172.21.15.151:0/688157424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 msgr2=0x7f30a4144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/688157424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f309c0099b0 tx=0x7f309c009250 comp rx=0 tx=0).stop 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 -- 172.21.15.151:0/688157424 shutdown_connections 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.675+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/688157424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a40aa9e0 0x7f30a4144850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/688157424 >> v2:172.21.15.172:3300/0 conn(0x7f30a4139f20 0x7f30a40aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- 172.21.15.151:0/688157424 >> 172.21.15.151:0/688157424 conn(0x7f30a4070510 msgr2=0x7f30a4070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:18.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- 172.21.15.151:0/688157424 shutdown_connections 2023-09-30T02:02:18.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- 172.21.15.151:0/688157424 wait complete. 2023-09-30T02:02:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 Processor -- start 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- start start 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30a40aa9e0 0x7f30a4141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 0x7f30a40a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40a7c20 con 0x7f30a40aa9e0 2023-09-30T02:02:18.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40a88e0 con 0x7f30a40b0a80 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40a66e0 con 0x7f30a4139f20 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:49272/0 (socket says 172.21.15.151:49272) 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 -- 172.21.15.151:0/1607057062 learned_addr learned my addr 172.21.15.151:0/1607057062 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a37fe700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30a40aa9e0 0x7f30a4141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a2ffd700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 0x7f30a40a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 -- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 msgr2=0x7f30a40a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 0x7f30a40a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 -- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30a40aa9e0 msgr2=0x7f30a4141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30a40aa9e0 0x7f30a4141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 -- 172.21.15.151:0/1607057062 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0094f0 con 0x7f30a4139f20 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a2ffd700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 0x7f30a40a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a3fff700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c011420 con 0x7f30a4139f20 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c011580 con 0x7f30a4139f20 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a40a5580 con 0x7f30a4139f20 2023-09-30T02:02:18.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c011940 con 0x7f30a4139f20 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.679+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a414e040 con 0x7f30a4139f20 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f309c011cf0 con 0x7f30a4139f20 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a40008d0 con 0x7f30a4139f20 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f309b7fe700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 0x7f3084062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f309c07e090 con 0x7f30a4139f20 2023-09-30T02:02:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f30a37fe700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 0x7f3084062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:18.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.683+0000 7f30a37fe700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 0x7f3084062a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f308c003a80 tx=0x7f308c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:18.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.687+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f309c04e7d0 con 0x7f30a4139f20 2023-09-30T02:02:18.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f30a40a6bd0 con 0x7f30a4139f20 2023-09-30T02:02:18.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f309b7fe700 1 -- 172.21.15.151:0/1607057062 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f309c051df0 con 0x7f30a4139f20 2023-09-30T02:02:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:18.891 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:18.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 msgr2=0x7f3084062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 0x7f3084062a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f308c003a80 tx=0x7f308c007ec0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 msgr2=0x7f30a40a6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c017000 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 shutdown_connections 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3084060580 0x7f3084062a30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30a4139f20 0x7f30a40a6f20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30a40b0a80 0x7f30a40a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 --2- 172.21.15.151:0/1607057062 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30a40aa9e0 0x7f30a4141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 >> 172.21.15.151:0/1607057062 conn(0x7f30a4070510 msgr2=0x7f30a406e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 shutdown_connections 2023-09-30T02:02:18.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:18.887+0000 7f30a8e6c700 1 -- 172.21.15.151:0/1607057062 wait complete. 2023-09-30T02:02:21.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:22.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 Processor -- start 2023-09-30T02:02:22.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 -- start start 2023-09-30T02:02:22.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f5006e070 0x7f1f5006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:22.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f500abe30 con 0x7f1f500b0440 2023-09-30T02:02:22.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5006d690 con 0x7f1f500b0070 2023-09-30T02:02:22.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f55a52700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5006d970 con 0x7f1f5006e070 2023-09-30T02:02:22.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4f7fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1f500b0440 0x7f1f5006bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48168/0 (socket says 172.21.15.151:48168) 2023-09-30T02:02:22.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4f7fe700 1 -- 172.21.15.151:0/841385624 learned_addr learned my addr 172.21.15.151:0/841385624 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:22.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4e7fc700 1 --2- 172.21.15.151:0/841385624 >> v2:172.21.15.172:3300/0 conn(0x7f1f5006e070 0x7f1f5006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:22.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269265531 0 0) 0x7f1f500abe30 con 0x7f1f500b0440 2023-09-30T02:02:22.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f38003570 con 0x7f1f500b0440 2023-09-30T02:02:22.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4e7fc700 1 -- 172.21.15.151:0/841385624 >> v1:172.21.15.172:6789/0 conn(0x7f1f500b0070 legacy=0x7f1f5013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689611596 0 0) 0x7f1f38003570 con 0x7f1f500b0440 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4e7fc700 1 -- 172.21.15.151:0/841385624 >> v1:172.21.15.151:6789/0 conn(0x7f1f500b0440 legacy=0x7f1f5006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4e7fc700 1 -- 172.21.15.151:0/841385624 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5006dc50 con 0x7f1f5006e070 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.015+0000 7f1f4e7fc700 1 --2- 172.21.15.151:0/841385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f5006e070 0x7f1f5006ea00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e60 tx=0x7f1f40009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3d4d3ee64306b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f48001470 con 0x7f1f5006e070 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4000b850 con 0x7f1f5006e070 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f4dffb700 1 -- 172.21.15.151:0/841385624 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000bbf0 con 0x7f1f5006e070 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- 172.21.15.151:0/841385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f5006e070 msgr2=0x7f1f5006ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 --2- 172.21.15.151:0/841385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f5006e070 0x7f1f5006ea00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e60 tx=0x7f1f40009780 comp rx=0 tx=0).stop 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- 172.21.15.151:0/841385624 shutdown_connections 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 --2- 172.21.15.151:0/841385624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f5006e070 0x7f1f5006ea00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- 172.21.15.151:0/841385624 >> 172.21.15.151:0/841385624 conn(0x7f1f50070fe0 msgr2=0x7f1f500af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:22.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- 172.21.15.151:0/841385624 shutdown_connections 2023-09-30T02:02:22.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- 172.21.15.151:0/841385624 wait complete. 2023-09-30T02:02:22.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 Processor -- start 2023-09-30T02:02:22.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.019+0000 7f1f55a52700 1 -- start start 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500b0070 0x7f1f5013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f500b0440 0x7f1f500a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5013dba0 con 0x7f1f5006e070 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50137090 con 0x7f1f500b0070 2023-09-30T02:02:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5013e380 con 0x7f1f500b0440 2023-09-30T02:02:22.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:22.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57482/0 (socket says 172.21.15.151:57482) 2023-09-30T02:02:22.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 -- 172.21.15.151:0/2006425264 learned_addr learned my addr 172.21.15.151:0/2006425264 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:22.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4f7fe700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f500b0440 0x7f1f500a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:22.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4e7fc700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500b0070 0x7f1f5013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 -- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f500b0440 msgr2=0x7f1f500a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f500b0440 0x7f1f500a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 -- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500b0070 msgr2=0x7f1f5013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500b0070 0x7f1f5013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 -- 172.21.15.151:0/2006425264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f40009a20 con 0x7f1f5006e070 2023-09-30T02:02:22.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f4effd700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c590 tx=0x7f1f4400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44011590 con 0x7f1f5006e070 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f44011710 con 0x7f1f5006e070 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44011ad0 con 0x7f1f5006e070 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f500a65c0 con 0x7f1f5006e070 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f500a9230 con 0x7f1f5006e070 2023-09-30T02:02:22.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.023+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f500670a0 con 0x7f1f5006e070 2023-09-30T02:02:22.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.027+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1f4401b3b0 con 0x7f1f5006e070 2023-09-30T02:02:22.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.027+0000 7f1f36ffd700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 0x7f1f3c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:22.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.027+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f1f4407b980 con 0x7f1f5006e070 2023-09-30T02:02:22.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.027+0000 7f1f4e7fc700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 0x7f1f3c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:22.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.027+0000 7f1f4e7fc700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 0x7f1f3c062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f50137c30 tx=0x7f1f40019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:22.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.031+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1f4404c160 con 0x7f1f5006e070 2023-09-30T02:02:22.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.227+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1f5006e4e0 con 0x7f1f5006e070 2023-09-30T02:02:22.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.227+0000 7f1f36ffd700 1 -- 172.21.15.151:0/2006425264 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1f4404f780 con 0x7f1f5006e070 2023-09-30T02:02:22.233 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:22.233 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 msgr2=0x7f1f3c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 0x7f1f3c062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f50137c30 tx=0x7f1f40019000 comp rx=0 tx=0).stop 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 msgr2=0x7f1f500a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c590 tx=0x7f1f4400c8a0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 shutdown_connections 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1f3c060550 0x7f1f3c062a00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f500b0440 0x7f1f500a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f500b0070 0x7f1f5013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 --2- 172.21.15.151:0/2006425264 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f5006e070 0x7f1f500a5f40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:22.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 >> 172.21.15.151:0/2006425264 conn(0x7f1f50070fe0 msgr2=0x7f1f500aa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:22.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 shutdown_connections 2023-09-30T02:02:22.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:22.231+0000 7f1f55a52700 1 -- 172.21.15.151:0/2006425264 wait complete. 2023-09-30T02:02:25.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:25.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 Processor -- start 2023-09-30T02:02:25.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- start start 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 0x7f530006d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530006c4b0 con 0x7f530006db80 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530006c720 con 0x7f53000b3f60 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530006ca00 con 0x7f53000b3b90 2023-09-30T02:02:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:57668/0 (socket says 172.21.15.151:57668) 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 -- 172.21.15.151:0/2835276429 learned_addr learned my addr 172.21.15.151:0/2835276429 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53049ef700 1 --2- 172.21.15.151:0/2835276429 >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 0x7f530006d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 -- 172.21.15.151:0/2835276429 >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 msgr2=0x7f530006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 --2- 172.21.15.151:0/2835276429 >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 0x7f530006d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 -- 172.21.15.151:0/2835276429 >> v1:172.21.15.151:6789/0 conn(0x7f530006db80 legacy=0x7f53000ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 -- 172.21.15.151:0/2835276429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530006b810 con 0x7f53000b3b90 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53049ef700 1 --2- 172.21.15.151:0/2835276429 >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 0x7f530006d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53051f0700 1 --2- 172.21.15.151:0/2835276429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009f20 tx=0x7f52e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623953293a21b1d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f52fffff700 1 -- 172.21.15.151:0/2835276429 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800e070 con 0x7f53000b3b90 2023-09-30T02:02:25.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f52fffff700 1 -- 172.21.15.151:0/2835276429 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e800ba80 con 0x7f53000b3b90 2023-09-30T02:02:25.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f52fffff700 1 -- 172.21.15.151:0/2835276429 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80092e0 con 0x7f53000b3b90 2023-09-30T02:02:25.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- 172.21.15.151:0/2835276429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 msgr2=0x7f530006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 --2- 172.21.15.151:0/2835276429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009f20 tx=0x7f52e8009840 comp rx=0 tx=0).stop 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- 172.21.15.151:0/2835276429 shutdown_connections 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 --2- 172.21.15.151:0/2835276429 >> v2:172.21.15.151:3300/0 conn(0x7f53000b3f60 0x7f530006d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 --2- 172.21.15.151:0/2835276429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f53000b3b90 0x7f530006d100 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- 172.21.15.151:0/2835276429 >> 172.21.15.151:0/2835276429 conn(0x7f5300063990 msgr2=0x7f5300070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- 172.21.15.151:0/2835276429 shutdown_connections 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.354+0000 7f53061f2700 1 -- 172.21.15.151:0/2835276429 wait complete. 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 Processor -- start 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- start start 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53000b3b90 0x7f530011b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53000b3f60 0x7f5300150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000aad80 con 0x7f53000b3b90 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000ac5e0 con 0x7f53000b3f60 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000ab930 con 0x7f530006db80 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53049ef700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53000b3b90 0x7f530011b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:57674/0 (socket says 172.21.15.151:57674) 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 -- 172.21.15.151:0/2438076003 learned_addr learned my addr 172.21.15.151:0/2438076003 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53059f1700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53000b3f60 0x7f5300150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 -- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53000b3f60 msgr2=0x7f5300150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53000b3f60 0x7f5300150410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 -- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53000b3b90 msgr2=0x7f530011b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53000b3b90 0x7f530011b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 -- 172.21.15.151:0/2438076003 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009ae0 con 0x7f530006db80 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53051f0700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f530006cb40 tx=0x7f52e8018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8007000 con 0x7f530006db80 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5300150950 con 0x7f530006db80 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5300150e70 con 0x7f530006db80 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e8007820 con 0x7f530006db80 2023-09-30T02:02:25.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.358+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8007be0 con 0x7f530006db80 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e40050b0 con 0x7f530006db80 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f52e8007d40 con 0x7f530006db80 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f52fd7fa700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 0x7f52e0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f52e807e2e0 con 0x7f530006db80 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f53049ef700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 0x7f52e0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:25.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.362+0000 7f53049ef700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 0x7f52e0062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f530013cb50 tx=0x7f52f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:25.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.366+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f52e804eac0 con 0x7f530006db80 2023-09-30T02:02:25.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.562+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f52e4005360 con 0x7f530006db80 2023-09-30T02:02:25.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.566+0000 7f52fd7fa700 1 -- 172.21.15.151:0/2438076003 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f52e80520e0 con 0x7f530006db80 2023-09-30T02:02:25.571 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:25.571 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 msgr2=0x7f52e0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 0x7f52e0062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f530013cb50 tx=0x7f52f8008040 comp rx=0 tx=0).stop 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 msgr2=0x7f530013e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f530006cb40 tx=0x7f52e8018000 comp rx=0 tx=0).stop 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 shutdown_connections 2023-09-30T02:02:25.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f52e0060550 0x7f52e0062a00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53000b3f60 0x7f5300150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53000b3b90 0x7f530011b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 --2- 172.21.15.151:0/2438076003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f530006db80 0x7f530013e670 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 >> 172.21.15.151:0/2438076003 conn(0x7f5300063990 msgr2=0x7f5300143010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 shutdown_connections 2023-09-30T02:02:25.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:25.570+0000 7f53061f2700 1 -- 172.21.15.151:0/2438076003 wait complete. 2023-09-30T02:02:28.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:28.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 Processor -- start 2023-09-30T02:02:28.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- start start 2023-09-30T02:02:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0ac330 con 0x7f90cc13b530 2023-09-30T02:02:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0adbc0 con 0x7f90cc13b120 2023-09-30T02:02:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0ae9c0 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:57680/0 (socket says 172.21.15.151:57680) 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 -- 172.21.15.151:0/3635075512 learned_addr learned my addr 172.21.15.151:0/3635075512 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 -- 172.21.15.151:0/3635075512 >> v1:172.21.15.172:6789/0 conn(0x7f90cc13b120 legacy=0x7f90cc0aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 -- 172.21.15.151:0/3635075512 >> v1:172.21.15.151:6789/0 conn(0x7f90cc13b530 legacy=0x7f90cc145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 -- 172.21.15.151:0/3635075512 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0ade30 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d1e5b700 1 --2- 172.21.15.151:0/3635075512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90c80099e0 tx=0x7f90c8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7a5c8438b9ac63 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:28.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d0e59700 1 -- 172.21.15.151:0/3635075512 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800b860 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d0e59700 1 -- 172.21.15.151:0/3635075512 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c800b9c0 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d0e59700 1 -- 172.21.15.151:0/3635075512 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800bd60 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- 172.21.15.151:0/3635075512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 msgr2=0x7f90cc0acaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/3635075512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90c80099e0 tx=0x7f90c8009250 comp rx=0 tx=0).stop 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- 172.21.15.151:0/3635075512 shutdown_connections 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/3635075512 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc0ae5f0 0x7f90cc0acaa0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- 172.21.15.151:0/3635075512 >> 172.21.15.151:0/3635075512 conn(0x7f90cc070510 msgr2=0x7f90cc0708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- 172.21.15.151:0/3635075512 shutdown_connections 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.698+0000 7f90d2e5d700 1 -- 172.21.15.151:0/3635075512 wait complete. 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 Processor -- start 2023-09-30T02:02:28.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- start start 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90cc0ae5f0 0x7f90cc14e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 0x7f90cc0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc14e760 con 0x7f90cc13b530 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0a47e0 con 0x7f90cc0ae5f0 2023-09-30T02:02:28.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0a68e0 con 0x7f90cc13b120 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d265c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 0x7f90cc0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d265c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:57528/0 (socket says 172.21.15.151:57528) 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d265c700 1 -- 172.21.15.151:0/2595005453 learned_addr learned my addr 172.21.15.151:0/2595005453 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d1e5b700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90cc0ae5f0 0x7f90cc14e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 -- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90cc0ae5f0 msgr2=0x7f90cc14e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90cc0ae5f0 0x7f90cc14e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 -- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 msgr2=0x7f90cc0a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 -- 172.21.15.151:0/2595005453 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80094f0 con 0x7f90cc13b120 2023-09-30T02:02:28.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d265c700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d165a700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 0x7f90cc0a63a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c590 tx=0x7f90bc007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc011590 con 0x7f90cc13b120 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc0a83c0 con 0x7f90cc13b120 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc011710 con 0x7f90cc13b120 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc011ad0 con 0x7f90cc13b120 2023-09-30T02:02:28.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.702+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc1359f0 con 0x7f90cc13b120 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc0a6a50 con 0x7f90cc13b120 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f90bc01b3b0 con 0x7f90cc13b120 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90c1ffb700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 0x7f90b0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f90bc07b9d0 con 0x7f90cc13b120 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90d1e5b700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 0x7f90b0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:28.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.706+0000 7f90d1e5b700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 0x7f90b0062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90c80099e0 tx=0x7f90c8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:28.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.714+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f90bc04c1b0 con 0x7f90cc13b120 2023-09-30T02:02:28.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.906+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f90cc0008d0 con 0x7f90cc13b120 2023-09-30T02:02:28.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.906+0000 7f90c1ffb700 1 -- 172.21.15.151:0/2595005453 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f90bc017080 con 0x7f90cc13b120 2023-09-30T02:02:28.914 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:28.914 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:28.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 msgr2=0x7f90b0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 0x7f90b0062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90c80099e0 tx=0x7f90c8017000 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 msgr2=0x7f90cc0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 0x7f90cc0a63a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c590 tx=0x7f90bc007ec0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 shutdown_connections 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90b0060550 0x7f90b0062a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13b530 0x7f90cc0a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc13b120 0x7f90cc0a63a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 --2- 172.21.15.151:0/2595005453 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90cc0ae5f0 0x7f90cc14e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 >> 172.21.15.151:0/2595005453 conn(0x7f90cc070510 msgr2=0x7f90cc13ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 shutdown_connections 2023-09-30T02:02:28.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:28.910+0000 7f90d2e5d700 1 -- 172.21.15.151:0/2595005453 wait complete. 2023-09-30T02:02:31.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:32.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 Processor -- start 2023-09-30T02:02:32.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 -- start start 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2b5006af50 0x7f2b501399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 0x7f2b500a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5006d8b0 con 0x7f2b5006af50 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5006db70 con 0x7f2b5006e000 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b548f6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5006c3a0 con 0x7f2b5006b320 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:35880/0 (socket says 172.21.15.151:35880) 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/2792848394 learned_addr learned my addr 172.21.15.151:0/2792848394 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4e59c700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.151:3300/0 conn(0x7f2b5006af50 0x7f2b501399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4ed9d700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 0x7f2b500a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/2792848394 >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 msgr2=0x7f2b500a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 0x7f2b500a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/2792848394 >> v2:172.21.15.151:3300/0 conn(0x7f2b5006af50 msgr2=0x7f2b501399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.151:3300/0 conn(0x7f2b5006af50 0x7f2b501399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/2792848394 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5006ced0 con 0x7f2b5006b320 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4ed9d700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 0x7f2b500a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/2792848394 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2b5013aa60 tx=0x7f2b40009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3536d3c071bd8a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4d59a700 1 -- 172.21.15.151:0/2792848394 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000b860 con 0x7f2b5006b320 2023-09-30T02:02:32.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4d59a700 1 -- 172.21.15.151:0/2792848394 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4000b9c0 con 0x7f2b5006b320 2023-09-30T02:02:32.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.038+0000 7f2b4d59a700 1 -- 172.21.15.151:0/2792848394 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000bd60 con 0x7f2b5006b320 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- 172.21.15.151:0/2792848394 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 msgr2=0x7f2b5006be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- 172.21.15.151:0/2792848394 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2b5013aa60 tx=0x7f2b40009250 comp rx=0 tx=0).stop 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- 172.21.15.151:0/2792848394 shutdown_connections 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.172:3300/0 conn(0x7f2b5006e000 0x7f2b500a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- 172.21.15.151:0/2792848394 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006b320 0x7f2b5006be60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- 172.21.15.151:0/2792848394 >> v2:172.21.15.151:3300/0 conn(0x7f2b5006af50 0x7f2b501399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- 172.21.15.151:0/2792848394 >> 172.21.15.151:0/2792848394 conn(0x7f2b50070fe0 msgr2=0x7f2b500af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- 172.21.15.151:0/2792848394 shutdown_connections 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- 172.21.15.151:0/2792848394 wait complete. 2023-09-30T02:02:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 Processor -- start 2023-09-30T02:02:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- start start 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 0x7f2b500a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006e000 0x7f2b5013d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5013c6a0 con 0x7f2b5006b320 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b500a7b80 con 0x7f2b5006af50 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b548f6700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b500a4800 con 0x7f2b5006e000 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b4dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b4dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39026/0 (socket says 172.21.15.151:39026) 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/3366888890 learned_addr learned my addr 172.21.15.151:0/3366888890 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.042+0000 7f2b4ed9d700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006e000 0x7f2b5013d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4e59c700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 0x7f2b500a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006e000 msgr2=0x7f2b5013d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006e000 0x7f2b5013d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 msgr2=0x7f2b500a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 0x7f2b500a5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 -- 172.21.15.151:0/3366888890 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400094f0 con 0x7f2b5006b320 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4e59c700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 0x7f2b500a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b4dd9b700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40024440 con 0x7f2b5006b320 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b400245a0 con 0x7f2b5006b320 2023-09-30T02:02:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b500a68f0 con 0x7f2b5006b320 2023-09-30T02:02:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40024960 con 0x7f2b5006b320 2023-09-30T02:02:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b500a6680 con 0x7f2b5006b320 2023-09-30T02:02:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b500a7cc0 con 0x7f2b5006b320 2023-09-30T02:02:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.046+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2b40024d50 con 0x7f2b5006b320 2023-09-30T02:02:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.050+0000 7f2b3e7fc700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 0x7f2b30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:32.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.050+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2b4007e1b0 con 0x7f2b5006b320 2023-09-30T02:02:32.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.050+0000 7f2b4e59c700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 0x7f2b30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:32.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.050+0000 7f2b4e59c700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 0x7f2b30062a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b44006fd0 tx=0x7f2b44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.058+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2b500a7cc0 con 0x7f2b5006b320 2023-09-30T02:02:32.258 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2b500a7cc0 con 0x7f2b5006b320 2023-09-30T02:02:32.259 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b3e7fc700 1 -- 172.21.15.151:0/3366888890 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2b500a7cc0 con 0x7f2b5006b320 2023-09-30T02:02:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:32.259 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 msgr2=0x7f2b30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 0x7f2b30062a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b44006fd0 tx=0x7f2b44008040 comp rx=0 tx=0).stop 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 msgr2=0x7f2b5013cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40017000 comp rx=0 tx=0).stop 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 shutdown_connections 2023-09-30T02:02:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b30060550 0x7f2b30062a00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b5006e000 0x7f2b5013d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b5006b320 0x7f2b5013cb30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 --2- 172.21.15.151:0/3366888890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b5006af50 0x7f2b500a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 >> 172.21.15.151:0/3366888890 conn(0x7f2b50070fe0 msgr2=0x7f2b500af190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 shutdown_connections 2023-09-30T02:02:32.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:32.254+0000 7f2b548f6700 1 -- 172.21.15.151:0/3366888890 wait complete. 2023-09-30T02:02:35.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:35.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 Processor -- start 2023-09-30T02:02:35.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- start start 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f718806e470 0x7f7188137f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7188067fa0 0x7f7188138490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880acd90 con 0x7f718813e8d0 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718806e840 con 0x7f7188067fa0 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71880ad000 con 0x7f718806e470 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7188067fa0 0x7f7188138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7188067fa0 0x7f7188138490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39040/0 (socket says 172.21.15.151:39040) 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 -- 172.21.15.151:0/2063705269 learned_addr learned my addr 172.21.15.151:0/2063705269 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718659c700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f718806e470 0x7f7188137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 -- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f718806e470 msgr2=0x7f7188137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f718806e470 0x7f7188137f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 -- 172.21.15.151:0/2063705269 >> v1:172.21.15.172:6789/0 conn(0x7f718813e8d0 legacy=0x7f718813ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 -- 172.21.15.151:0/2063705269 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880abaf0 con 0x7f7188067fa0 2023-09-30T02:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f717dd9b700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7188067fa0 0x7f7188138490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f71700099b0 tx=0x7f7170009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7c8ca72dfbc35c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f7185d9b700 1 -- 172.21.15.151:0/2063705269 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000b820 con 0x7f7188067fa0 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f7185d9b700 1 -- 172.21.15.151:0/2063705269 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717000b980 con 0x7f7188067fa0 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f7185d9b700 1 -- 172.21.15.151:0/2063705269 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000bd40 con 0x7f7188067fa0 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7188067fa0 msgr2=0x7f7188138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7188067fa0 0x7f7188138490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f71700099b0 tx=0x7f7170009250 comp rx=0 tx=0).stop 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- 172.21.15.151:0/2063705269 shutdown_connections 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7188067fa0 0x7f7188138490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 --2- 172.21.15.151:0/2063705269 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f718806e470 0x7f7188137f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- 172.21.15.151:0/2063705269 >> 172.21.15.151:0/2063705269 conn(0x7f7188070510 msgr2=0x7f7188070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- 172.21.15.151:0/2063705269 shutdown_connections 2023-09-30T02:02:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.386+0000 7f718759e700 1 -- 172.21.15.151:0/2063705269 wait complete. 2023-09-30T02:02:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 Processor -- start 2023-09-30T02:02:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- start start 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7188067fa0 0x7f71880b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 0x7f718806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718806d340 con 0x7f718806e470 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718806cf40 con 0x7f718813e8d0 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718814db70 con 0x7f7188067fa0 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718659c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7188067fa0 0x7f71880b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39050/0 (socket says 172.21.15.151:39050) 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 -- 172.21.15.151:0/2404877187 learned_addr learned my addr 172.21.15.151:0/2404877187 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f7186d9d700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 0x7f718806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 -- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7188067fa0 msgr2=0x7f71880b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7188067fa0 0x7f71880b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 -- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 msgr2=0x7f718806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 0x7f718806d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 -- 172.21.15.151:0/2404877187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700094f0 con 0x7f718806e470 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f7186d9d700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 0x7f718806d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717dd9b700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7188071540 tx=0x7f7170016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:35.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000b820 con 0x7f718806e470 2023-09-30T02:02:35.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170020070 con 0x7f718806e470 2023-09-30T02:02:35.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718814dd30 con 0x7f718806e470 2023-09-30T02:02:35.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170011da0 con 0x7f718806e470 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.390+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718806cc70 con 0x7f718806e470 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718806d4f0 con 0x7f718806e470 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7170022030 con 0x7f718806e470 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f717effd700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 0x7f7168062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f717007e350 con 0x7f718806e470 2023-09-30T02:02:35.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f718659c700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 0x7f7168062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:35.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.394+0000 7f718659c700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 0x7f7168062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7178006fd0 tx=0x7f7178008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:35.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.402+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f718806d4f0 con 0x7f718806e470 2023-09-30T02:02:35.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.602+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f718806d100 con 0x7f718806e470 2023-09-30T02:02:35.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.602+0000 7f717effd700 1 -- 172.21.15.151:0/2404877187 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f717004eb30 con 0x7f718806e470 2023-09-30T02:02:35.610 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:35.610 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 msgr2=0x7f7168062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 0x7f7168062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7178006fd0 tx=0x7f7178008040 comp rx=0 tx=0).stop 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 msgr2=0x7f718814d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7188071540 tx=0x7f7170016000 comp rx=0 tx=0).stop 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 shutdown_connections 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7168060550 0x7f7168062a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f718813e8d0 0x7f718806d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f718806e470 0x7f718814d060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 --2- 172.21.15.151:0/2404877187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7188067fa0 0x7f71880b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 >> 172.21.15.151:0/2404877187 conn(0x7f7188070510 msgr2=0x7f71881373c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 shutdown_connections 2023-09-30T02:02:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:35.606+0000 7f718759e700 1 -- 172.21.15.151:0/2404877187 wait complete. 2023-09-30T02:02:38.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:38.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.726+0000 7fd55072c700 1 Processor -- start 2023-09-30T02:02:38.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.726+0000 7fd55072c700 1 -- start start 2023-09-30T02:02:38.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54806dca0 con 0x7fd54806c7f0 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480aafe0 con 0x7fd5480aac10 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480ad440 con 0x7fd54806e910 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54dcc7700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd54806c7f0 0x7fd548136de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:36824/0 (socket says 172.21.15.151:36824) 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:35922/0 (socket says 172.21.15.151:35922) 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54dcc7700 1 -- 172.21.15.151:0/711397057 learned_addr learned my addr 172.21.15.151:0/711397057 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 -- 172.21.15.151:0/711397057 >> v1:172.21.15.172:6789/0 conn(0x7fd5480aac10 legacy=0x7fd54806c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54d4c6700 1 -- 172.21.15.151:0/711397057 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047540423 0 0) 0x7fd54806dca0 con 0x7fd54806c7f0 2023-09-30T02:02:38.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 -- 172.21.15.151:0/711397057 >> v1:172.21.15.151:6789/0 conn(0x7fd54806c7f0 legacy=0x7fd548136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 -- 172.21.15.151:0/711397057 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54806ed90 con 0x7fd54806e910 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54ecc9700 1 --2- 172.21.15.151:0/711397057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd540009940 tx=0x7fd5400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae2b130a6356386 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54d4c6700 1 -- 172.21.15.151:0/711397057 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000b820 con 0x7fd54806e910 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54d4c6700 1 -- 172.21.15.151:0/711397057 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000b980 con 0x7fd54806e910 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd54d4c6700 1 -- 172.21.15.151:0/711397057 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000bd20 con 0x7fd54806e910 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- 172.21.15.151:0/711397057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 msgr2=0x7fd5481374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 --2- 172.21.15.151:0/711397057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd540009940 tx=0x7fd5400091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- 172.21.15.151:0/711397057 shutdown_connections 2023-09-30T02:02:38.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 --2- 172.21.15.151:0/711397057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd5481374f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- 172.21.15.151:0/711397057 >> 172.21.15.151:0/711397057 conn(0x7fd548070510 msgr2=0x7fd548070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:38.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- 172.21.15.151:0/711397057 shutdown_connections 2023-09-30T02:02:38.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.730+0000 7fd55072c700 1 -- 172.21.15.151:0/711397057 wait complete. 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 Processor -- start 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- start start 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54806c7f0 0x7fd54813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd54814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd548067f80 con 0x7fd5480aac10 2023-09-30T02:02:38.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480a6170 con 0x7fd54806c7f0 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480a63e0 con 0x7fd54806e910 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54dcc7700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd54814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:39058/0 (socket says 172.21.15.151:39058) 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 -- 172.21.15.151:0/1995981871 learned_addr learned my addr 172.21.15.151:0/1995981871 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54e4c8700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54806c7f0 0x7fd54813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:38.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 -- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 msgr2=0x7fd54814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd54814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 -- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54806c7f0 msgr2=0x7fd54813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54806c7f0 0x7fd54813a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 -- 172.21.15.151:0/1995981871 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540009420 con 0x7fd5480aac10 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54dcc7700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd54814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd54ecc9700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd540009860 tx=0x7fd540017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54001f070 con 0x7fd5480aac10 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540002f70 con 0x7fd5480aac10 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd548067370 con 0x7fd5480aac10 2023-09-30T02:02:38.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540025650 con 0x7fd5480aac10 2023-09-30T02:02:38.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.734+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54814ec50 con 0x7fd5480aac10 2023-09-30T02:02:38.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5480680f0 con 0x7fd5480aac10 2023-09-30T02:02:38.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd540023070 con 0x7fd5480aac10 2023-09-30T02:02:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd5467fc700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 0x7fd534062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd54007db80 con 0x7fd5480aac10 2023-09-30T02:02:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd54e4c8700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 0x7fd534062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:38.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.738+0000 7fd54e4c8700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 0x7fd534062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0039e0 tx=0x7fd53c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:38.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.750+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd54004e2c0 con 0x7fd5480aac10 2023-09-30T02:02:38.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.934+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd54814ec50 con 0x7fd5480aac10 2023-09-30T02:02:38.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.934+0000 7fd5467fc700 1 -- 172.21.15.151:0/1995981871 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd540015100 con 0x7fd5480aac10 2023-09-30T02:02:38.941 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:38.941 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 msgr2=0x7fd534062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 0x7fd534062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0039e0 tx=0x7fd53c007ec0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 msgr2=0x7fd54814e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd540009860 tx=0x7fd540017000 comp rx=0 tx=0).stop 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 shutdown_connections 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fd534060550 0x7fd534062a00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd5480aac10 0x7fd54814e4f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd54806e910 0x7fd54814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 --2- 172.21.15.151:0/1995981871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd54806c7f0 0x7fd54813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:38.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 >> 172.21.15.151:0/1995981871 conn(0x7fd548070510 msgr2=0x7fd5480abb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:38.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 shutdown_connections 2023-09-30T02:02:38.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:38.938+0000 7fd55072c700 1 -- 172.21.15.151:0/1995981871 wait complete. 2023-09-30T02:02:41.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:42.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 Processor -- start 2023-09-30T02:02:42.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 -- start start 2023-09-30T02:02:42.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ed00b1ad0 0x7f6ed00ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed00add40 con 0x7f6ed006ab80 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed00ad000 con 0x7f6ed00b1ad0 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.062+0000 7f6ed4938700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00ae020 con 0x7f6ed006a230 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43698/0 (socket says 172.21.15.151:43698) 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 -- 172.21.15.151:0/1022945596 learned_addr learned my addr 172.21.15.151:0/1022945596 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ecf7fe700 1 --2- 172.21.15.151:0/1022945596 >> v2:172.21.15.151:3300/0 conn(0x7f6ed00b1ad0 0x7f6ed00ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 -- 172.21.15.151:0/1022945596 >> v2:172.21.15.151:3300/0 conn(0x7f6ed00b1ad0 msgr2=0x7f6ed00ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 --2- 172.21.15.151:0/1022945596 >> v2:172.21.15.151:3300/0 conn(0x7f6ed00b1ad0 0x7f6ed00ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 -- 172.21.15.151:0/1022945596 >> v1:172.21.15.172:6789/0 conn(0x7f6ed006ab80 legacy=0x7f6ed006d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:42.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 -- 172.21.15.151:0/1022945596 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed00b1ea0 con 0x7f6ed006a230 2023-09-30T02:02:42.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6eceffd700 1 --2- 172.21.15.151:0/1022945596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00099b0 tx=0x7f6ec0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904f5a22d21256b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:42.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ece7fc700 1 -- 172.21.15.151:0/1022945596 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0009ea0 con 0x7f6ed006a230 2023-09-30T02:02:42.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ece7fc700 1 -- 172.21.15.151:0/1022945596 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec000b820 con 0x7f6ed006a230 2023-09-30T02:02:42.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ece7fc700 1 -- 172.21.15.151:0/1022945596 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec000bbc0 con 0x7f6ed006a230 2023-09-30T02:02:42.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- 172.21.15.151:0/1022945596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 msgr2=0x7f6ed006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- 172.21.15.151:0/1022945596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00099b0 tx=0x7f6ec0009250 comp rx=0 tx=0).stop 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- 172.21.15.151:0/1022945596 shutdown_connections 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- 172.21.15.151:0/1022945596 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed006a640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- 172.21.15.151:0/1022945596 >> v2:172.21.15.151:3300/0 conn(0x7f6ed00b1ad0 0x7f6ed00ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- 172.21.15.151:0/1022945596 >> 172.21.15.151:0/1022945596 conn(0x7f6ed0070510 msgr2=0x7f6ed0070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- 172.21.15.151:0/1022945596 shutdown_connections 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- 172.21.15.151:0/1022945596 wait complete. 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 Processor -- start 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 -- start start 2023-09-30T02:02:42.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed006ab80 0x7f6ed0139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.066+0000 7f6ed4938700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00ab2f0 con 0x7f6ed00b1ad0 2023-09-30T02:02:42.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00ab560 con 0x7f6ed006ab80 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00aa290 con 0x7f6ed006a230 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43700/0 (socket says 172.21.15.151:43700) 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:59942/0 (socket says 172.21.15.151:59942) 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 -- 172.21.15.151:0/734249904 learned_addr learned my addr 172.21.15.151:0/734249904 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:42.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6eceffd700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed006ab80 0x7f6ed0139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 -- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed006ab80 msgr2=0x7f6ed0139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed006ab80 0x7f6ed0139aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 -- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 msgr2=0x7f6ed0139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 -- 172.21.15.151:0/734249904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec00094f0 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecffff700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ecf7fe700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400e930 tx=0x7f6ec40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4013680 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec40137e0 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4013b80 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed013a520 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed00a9f50 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.074+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed00008d0 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.078+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6ec400c3e0 con 0x7f6ed006a230 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.078+0000 7f6eb77fe700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 0x7f6ebc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.078+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6ec407c790 con 0x7f6ed006a230 2023-09-30T02:02:42.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.078+0000 7f6eceffd700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 0x7f6ebc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:42.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.078+0000 7f6eceffd700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 0x7f6ebc062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00099b0 tx=0x7f6ec0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.082+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ec404ced0 con 0x7f6ed006a230 2023-09-30T02:02:42.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.274+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ed006d8b0 con 0x7f6ed006a230 2023-09-30T02:02:42.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.274+0000 7f6eb77fe700 1 -- 172.21.15.151:0/734249904 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6ec40504f0 con 0x7f6ed006a230 2023-09-30T02:02:42.280 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:42.281 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:42.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 msgr2=0x7f6ebc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:42.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 0x7f6ebc062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00099b0 tx=0x7f6ec0016000 comp rx=0 tx=0).stop 2023-09-30T02:02:42.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 msgr2=0x7f6ed00a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:42.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400e930 tx=0x7f6ec40091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 shutdown_connections 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f6ebc060550 0x7f6ebc062a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ed00b1ad0 0x7f6ed0139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6ed006ab80 0x7f6ed0139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 --2- 172.21.15.151:0/734249904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ed006a230 0x7f6ed00a91a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 >> 172.21.15.151:0/734249904 conn(0x7f6ed0070510 msgr2=0x7f6ed0140ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 shutdown_connections 2023-09-30T02:02:42.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:42.278+0000 7f6ed4938700 1 -- 172.21.15.151:0/734249904 wait complete. 2023-09-30T02:02:45.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.406+0000 7f534a0e3700 1 Processor -- start 2023-09-30T02:02:45.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.406+0000 7f534a0e3700 1 -- start start 2023-09-30T02:02:45.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 0x7f534406d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440b0d30 con 0x7f534406b3d0 2023-09-30T02:02:45.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534406c690 con 0x7f5344140c10 2023-09-30T02:02:45.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440ae490 con 0x7f53440b0960 2023-09-30T02:02:45.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f533bfff700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f534406b3d0 0x7f534406d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58414/0 (socket says 172.21.15.151:58414) 2023-09-30T02:02:45.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f533bfff700 1 -- 172.21.15.151:0/2655325630 learned_addr learned my addr 172.21.15.151:0/2655325630 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:45.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 0x7f534406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 -- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 msgr2=0x7f534406d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2023-09-30T02:02:45.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 0x7f534406d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2023-09-30T02:02:45.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 0x7f534406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165370916 0 0) 0x7f53440b0d30 con 0x7f534406b3d0 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5330003570 con 0x7f534406b3d0 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718876139 0 0) 0x7f534406c690 con 0x7f5344140c10 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53440b0d30 con 0x7f5344140c10 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855994783 0 0) 0x7f5330003570 con 0x7f534406b3d0 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 msgr2=0x7f534406d270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 --2- 172.21.15.151:0/2655325630 >> v2:172.21.15.172:3300/0 conn(0x7f53440b0960 0x7f534406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 >> v1:172.21.15.172:6789/0 conn(0x7f5344140c10 legacy=0x7f5344141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534406c970 con 0x7f534406b3d0 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f532c001470 con 0x7f534406b3d0 2023-09-30T02:02:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 >> v1:172.21.15.151:6789/0 conn(0x7f534406b3d0 legacy=0x7f534406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 0x7f5330005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53300037b0 0x7f5330003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53300041b0 0x7f533000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0015b0 con 0x7f53300041b0 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533000c1b0 con 0x7f53300037b0 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533000c370 con 0x7f5330004c80 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 0x7f5330005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5343fff700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53300041b0 0x7f533000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f533bfff700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53300037b0 0x7f5330003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 -- 172.21.15.151:0/2655325630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53300037b0 msgr2=0x7f5330003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:45.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53300037b0 0x7f5330003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 -- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53300041b0 msgr2=0x7f533000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53300041b0 0x7f533000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 -- 172.21.15.151:0/2655325630 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330003570 con 0x7f5330004c80 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5343fff700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53300041b0 0x7f533000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5343fff700 1 -- 172.21.15.151:0/2655325630 reap_dead start 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f53437fe700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 0x7f5330005050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f533400bc20 tx=0x7f5334008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=526f1be02f6309e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533400d6a0 con 0x7f5330004c80 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533400d820 con 0x7f5330004c80 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f5342ffd700 1 -- 172.21.15.151:0/2655325630 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533400dbe0 con 0x7f5330004c80 2023-09-30T02:02:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 -- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 msgr2=0x7f5330005050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.410+0000 7f534a0e3700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 0x7f5330005050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f533400bc20 tx=0x7f5334008040 comp rx=0 tx=0).stop 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- 172.21.15.151:0/2655325630 shutdown_connections 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 --2- 172.21.15.151:0/2655325630 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5330004c80 0x7f5330005050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- 172.21.15.151:0/2655325630 >> 172.21.15.151:0/2655325630 conn(0x7f5344070510 msgr2=0x7f5344070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- 172.21.15.151:0/2655325630 shutdown_connections 2023-09-30T02:02:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- 172.21.15.151:0/2655325630 wait complete. 2023-09-30T02:02:45.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 Processor -- start 2023-09-30T02:02:45.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- start start 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 0x7f534414cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 0x7f53440ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534414db30 con 0x7f5330004c80 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534414d1e0 con 0x7f534414d700 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f534a0e3700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534414a6f0 con 0x7f534411d670 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f53437fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 0x7f534414cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43720/0 (socket says 172.21.15.151:43720) 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 -- 172.21.15.151:0/3554617284 learned_addr learned my addr 172.21.15.151:0/3554617284 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:45.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f533bfff700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 0x7f53440ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 -- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 msgr2=0x7f53440ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 0x7f53440ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 -- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 msgr2=0x7f534414cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 0x7f534414cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 -- 172.21.15.151:0/3554617284 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440ad2b0 con 0x7f534411d670 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f53437fe700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 0x7f534414cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f533bfff700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 0x7f53440ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.414+0000 7f5343fff700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f53440afd80 tx=0x7f533c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c001080 con 0x7f534411d670 2023-09-30T02:02:45.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c00c7b0 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534411dae0 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c00cb70 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5344150690 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440670a0 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f533c0030d0 con 0x7f534411d670 2023-09-30T02:02:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 0x7f5324062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:45.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f533c07ca60 con 0x7f534411d670 2023-09-30T02:02:45.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.418+0000 7f53437fe700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 0x7f5324062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:45.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.422+0000 7f53437fe700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 0x7f5324062a30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f533400bfd0 tx=0x7f533401a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:45.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.426+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f533c04d1a0 con 0x7f534411d670 2023-09-30T02:02:45.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f534406b930 con 0x7f534411d670 2023-09-30T02:02:45.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f533b7fe700 1 -- 172.21.15.151:0/3554617284 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f533c0507c0 con 0x7f534411d670 2023-09-30T02:02:45.619 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:45.619 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 msgr2=0x7f5324062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 0x7f5324062a30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f533400bfd0 tx=0x7f533401a000 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 msgr2=0x7f5344150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f53440afd80 tx=0x7f533c009040 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 shutdown_connections 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5324060580 0x7f5324062a30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f534411d670 0x7f5344150150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f534414d700 0x7f53440ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 --2- 172.21.15.151:0/3554617284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5330004c80 0x7f534414cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:45.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 >> 172.21.15.151:0/3554617284 conn(0x7f5344070510 msgr2=0x7f53440708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:45.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 shutdown_connections 2023-09-30T02:02:45.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:45.618+0000 7f534a0e3700 1 -- 172.21.15.151:0/3554617284 wait complete. 2023-09-30T02:02:48.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:48.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.742+0000 7f0ce359e700 1 Processor -- start 2023-09-30T02:02:48.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- start start 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0ce4070740 0x7f0ce413d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ce40aae90 0x7f0ce413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40ab3e0 con 0x7f0ce41407c0 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce40ab630 con 0x7f0ce4070740 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce413d270 con 0x7f0ce40aae90 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0ce4070740 0x7f0ce413d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce2d9d700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0ce41407c0 0x7f0ce4140c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:58416/0 (socket says 172.21.15.151:58416) 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce2d9d700 1 -- 172.21.15.151:0/257321504 learned_addr learned my addr 172.21.15.151:0/257321504 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:48.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce1d9b700 1 --2- 172.21.15.151:0/257321504 >> v2:172.21.15.172:3300/0 conn(0x7f0ce40aae90 0x7f0ce413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce159a700 1 -- 172.21.15.151:0/257321504 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485310809 0 0) 0x7f0ce40ab3e0 con 0x7f0ce41407c0 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce159a700 1 -- 172.21.15.151:0/257321504 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd0003570 con 0x7f0ce41407c0 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 -- 172.21.15.151:0/257321504 >> v2:172.21.15.172:3300/0 conn(0x7f0ce40aae90 msgr2=0x7f0ce413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 --2- 172.21.15.151:0/257321504 >> v2:172.21.15.172:3300/0 conn(0x7f0ce40aae90 0x7f0ce413dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 -- 172.21.15.151:0/257321504 >> v1:172.21.15.151:6789/0 conn(0x7f0ce41407c0 legacy=0x7f0ce4140c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 -- 172.21.15.151:0/257321504 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4070b10 con 0x7f0ce4070740 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce259c700 1 --2- 172.21.15.151:0/257321504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce4070740 0x7f0ce413d6c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ce406b210 tx=0x7f0cd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5b6c2424c60b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce159a700 1 -- 172.21.15.151:0/257321504 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4011630 con 0x7f0ce4070740 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce159a700 1 -- 172.21.15.151:0/257321504 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd4011790 con 0x7f0ce4070740 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce159a700 1 -- 172.21.15.151:0/257321504 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4011b30 con 0x7f0ce4070740 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- 172.21.15.151:0/257321504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce4070740 msgr2=0x7f0ce413d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 --2- 172.21.15.151:0/257321504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce4070740 0x7f0ce413d6c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ce406b210 tx=0x7f0cd40091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- 172.21.15.151:0/257321504 shutdown_connections 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 --2- 172.21.15.151:0/257321504 >> v2:172.21.15.172:3300/0 conn(0x7f0ce40aae90 0x7f0ce413dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 --2- 172.21.15.151:0/257321504 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce4070740 0x7f0ce413d6c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- 172.21.15.151:0/257321504 >> 172.21.15.151:0/257321504 conn(0x7f0ce40b03f0 msgr2=0x7f0ce40b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- 172.21.15.151:0/257321504 shutdown_connections 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.746+0000 7f0ce359e700 1 -- 172.21.15.151:0/257321504 wait complete. 2023-09-30T02:02:48.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 Processor -- start 2023-09-30T02:02:48.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- start start 2023-09-30T02:02:48.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce40aae90 0x7f0ce414c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 0x7f0ce414ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce406db50 con 0x7f0ce40aae90 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce413ffe0 con 0x7f0ce41407c0 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41401a0 con 0x7f0ce4070740 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:43734/0 (socket says 172.21.15.151:43734) 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 -- 172.21.15.151:0/390896165 learned_addr learned my addr 172.21.15.151:0/390896165 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce1d9b700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce40aae90 0x7f0ce414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce2d9d700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 0x7f0ce414ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 -- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 msgr2=0x7f0ce414ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 0x7f0ce414ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 -- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce40aae90 msgr2=0x7f0ce414c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce40aae90 0x7f0ce414c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 -- 172.21.15.151:0/390896165 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4009420 con 0x7f0ce4070740 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce2d9d700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 0x7f0ce414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce259c700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009ed0 tx=0x7f0cd4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4000d10 con 0x7f0ce4070740 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd4011900 con 0x7f0ce4070740 2023-09-30T02:02:48.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4026430 con 0x7f0ce4070740 2023-09-30T02:02:48.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce406d630 con 0x7f0ce4070740 2023-09-30T02:02:48.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.750+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce406e180 con 0x7f0ce4070740 2023-09-30T02:02:48.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce406d900 con 0x7f0ce4070740 2023-09-30T02:02:48.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0cd4023030 con 0x7f0ce4070740 2023-09-30T02:02:48.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0cda7fc700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 0x7f0cc4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:48.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0cd407dc50 con 0x7f0ce4070740 2023-09-30T02:02:48.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0ce1d9b700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 0x7f0cc4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:48.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.754+0000 7f0ce1d9b700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 0x7f0cc4062a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0ce414a310 tx=0x7f0ccc0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:48.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.758+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0cd4083020 con 0x7f0ce4070740 2023-09-30T02:02:48.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.954+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0ce40008d0 con 0x7f0ce4070740 2023-09-30T02:02:48.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.954+0000 7f0cda7fc700 1 -- 172.21.15.151:0/390896165 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0cd404e4b0 con 0x7f0ce4070740 2023-09-30T02:02:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:48.958 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 msgr2=0x7f0cc4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 0x7f0cc4062a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0ce414a310 tx=0x7f0ccc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 msgr2=0x7f0ce413f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009ed0 tx=0x7f0cd4017000 comp rx=0 tx=0).stop 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 shutdown_connections 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0cc4060550 0x7f0cc4062a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce41407c0 0x7f0ce414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ce40aae90 0x7f0ce414c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 --2- 172.21.15.151:0/390896165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ce4070740 0x7f0ce413f500 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:48.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 >> 172.21.15.151:0/390896165 conn(0x7f0ce40b03f0 msgr2=0x7f0ce413ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:48.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 shutdown_connections 2023-09-30T02:02:48.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:48.958+0000 7f0ce359e700 1 -- 172.21.15.151:0/390896165 wait complete. 2023-09-30T02:02:51.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:52.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 Processor -- start 2023-09-30T02:02:52.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 -- start start 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efc740aaec0 0x7efc740aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efc740b0440 0x7efc7413b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74138dc0 con 0x7efc740b0070 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740abcb0 con 0x7efc740aaec0 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc7af1a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740abeb0 con 0x7efc740b0440 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efc740aaec0 0x7efc740aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efc740aaec0 0x7efc740aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53230/0 (socket says 172.21.15.151:53230) 2023-09-30T02:02:52.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 -- 172.21.15.151:0/3834868701 learned_addr learned my addr 172.21.15.151:0/3834868701 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc794b7700 1 --2- 172.21.15.151:0/3834868701 >> v2:172.21.15.172:3300/0 conn(0x7efc740b0440 0x7efc7413b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 -- 172.21.15.151:0/3834868701 >> v2:172.21.15.172:3300/0 conn(0x7efc740b0440 msgr2=0x7efc7413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 --2- 172.21.15.151:0/3834868701 >> v2:172.21.15.172:3300/0 conn(0x7efc740b0440 0x7efc7413b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 -- 172.21.15.151:0/3834868701 >> v1:172.21.15.172:6789/0 conn(0x7efc740b0070 legacy=0x7efc740a8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 -- 172.21.15.151:0/3834868701 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740ac190 con 0x7efc740aaec0 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc73fff700 1 --2- 172.21.15.151:0/3834868701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc740aa2d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc7406ed50 tx=0x7efc64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ea315364f10533 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc737fe700 1 -- 172.21.15.151:0/3834868701 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6400e070 con 0x7efc740aaec0 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc737fe700 1 -- 172.21.15.151:0/3834868701 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6400ba10 con 0x7efc740aaec0 2023-09-30T02:02:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.082+0000 7efc737fe700 1 -- 172.21.15.151:0/3834868701 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640092e0 con 0x7efc740aaec0 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- 172.21.15.151:0/3834868701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 msgr2=0x7efc740aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 --2- 172.21.15.151:0/3834868701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc740aa2d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc7406ed50 tx=0x7efc64009840 comp rx=0 tx=0).stop 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- 172.21.15.151:0/3834868701 shutdown_connections 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 --2- 172.21.15.151:0/3834868701 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc740aa2d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 --2- 172.21.15.151:0/3834868701 >> v2:172.21.15.172:3300/0 conn(0x7efc740b0440 0x7efc7413b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- 172.21.15.151:0/3834868701 >> 172.21.15.151:0/3834868701 conn(0x7efc74070fe0 msgr2=0x7efc740af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:52.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- 172.21.15.151:0/3834868701 shutdown_connections 2023-09-30T02:02:52.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- 172.21.15.151:0/3834868701 wait complete. 2023-09-30T02:02:52.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 Processor -- start 2023-09-30T02:02:52.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.086+0000 7efc7af1a700 1 -- start start 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 0x7efc7414aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc740b0440 0x7efc740a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740a7150 con 0x7efc740aaec0 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740a8640 con 0x7efc740b0440 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7414aff0 con 0x7efc740b0070 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc73fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 0x7efc7414aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53234/0 (socket says 172.21.15.151:53234) 2023-09-30T02:02:52.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc73fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 0x7efc7414aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52402/0 (socket says 172.21.15.151:52402) 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 -- 172.21.15.151:0/178445160 learned_addr learned my addr 172.21.15.151:0/178445160 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc794b7700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc740b0440 0x7efc740a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 -- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 msgr2=0x7efc7414aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 0x7efc7414aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 -- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc740b0440 msgr2=0x7efc740a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc740b0440 0x7efc740a4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 -- 172.21.15.151:0/178445160 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc64009ae0 con 0x7efc740aaec0 2023-09-30T02:02:52.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc78cb6700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c5d0 tx=0x7efc68007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68011590 con 0x7efc740aaec0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68011710 con 0x7efc740aaec0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7414b270 con 0x7efc740aaec0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68011ad0 con 0x7efc740aaec0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc740a5260 con 0x7efc740aaec0 2023-09-30T02:02:52.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.090+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc740670a0 con 0x7efc740aaec0 2023-09-30T02:02:52.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.094+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efc6801c460 con 0x7efc740aaec0 2023-09-30T02:02:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.094+0000 7efc70ff9700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 0x7efc60062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:52.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.094+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efc6807cde0 con 0x7efc740aaec0 2023-09-30T02:02:52.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.094+0000 7efc73fff700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 0x7efc60062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:52.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.094+0000 7efc73fff700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 0x7efc60062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc740ab300 tx=0x7efc64017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:52.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.102+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efc6804d5a0 con 0x7efc740aaec0 2023-09-30T02:02:52.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.294+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc740a4720 con 0x7efc740aaec0 2023-09-30T02:02:52.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.294+0000 7efc70ff9700 1 -- 172.21.15.151:0/178445160 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efc68050bc0 con 0x7efc740aaec0 2023-09-30T02:02:52.297 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:52.297 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 msgr2=0x7efc60062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 0x7efc60062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc740ab300 tx=0x7efc64017000 comp rx=0 tx=0).stop 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 msgr2=0x7efc7413d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c5d0 tx=0x7efc68007f60 comp rx=0 tx=0).stop 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 shutdown_connections 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7efc60060550 0x7efc60062a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc740b0440 0x7efc740a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc740b0070 0x7efc7414aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 --2- 172.21.15.151:0/178445160 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc740aaec0 0x7efc7413d270 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 >> 172.21.15.151:0/178445160 conn(0x7efc74070fe0 msgr2=0x7efc740af940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 shutdown_connections 2023-09-30T02:02:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:52.298+0000 7efc7af1a700 1 -- 172.21.15.151:0/178445160 wait complete. 2023-09-30T02:02:55.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:02:55.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 Processor -- start 2023-09-30T02:02:55.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 -- start start 2023-09-30T02:02:55.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 0x7f0e8c0aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8c138dc0 con 0x7f0e8c0aaec0 2023-09-30T02:02:55.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c0abcb0 con 0x7f0e8c0b0440 2023-09-30T02:02:55.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e913bd700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c0abeb0 con 0x7f0e8c0b0070 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53246/0 (socket says 172.21.15.151:53246) 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/1171611628 learned_addr learned my addr 172.21.15.151:0/1171611628 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8affd700 1 --2- 172.21.15.151:0/1171611628 >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 0x7f0e8c0aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/1171611628 >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 msgr2=0x7f0e8c0aa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/1171611628 >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 0x7f0e8c0aa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/1171611628 >> v1:172.21.15.172:6789/0 conn(0x7f0e8c0aaec0 legacy=0x7f0e8c0a8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/1171611628 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c0ac190 con 0x7f0e8c0b0440 2023-09-30T02:02:55.430 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8affd700 1 --2- 172.21.15.151:0/1171611628 >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 0x7f0e8c0aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:55.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/1171611628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0af8c0 tx=0x7f0e84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aa98928d63bedc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e89ffb700 1 -- 172.21.15.151:0/1171611628 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400b860 con 0x7f0e8c0b0440 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e89ffb700 1 -- 172.21.15.151:0/1171611628 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8400b9c0 con 0x7f0e8c0b0440 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.426+0000 7f0e89ffb700 1 -- 172.21.15.151:0/1171611628 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400bd80 con 0x7f0e8c0b0440 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- 172.21.15.151:0/1171611628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 msgr2=0x7f0e8c13b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 --2- 172.21.15.151:0/1171611628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0af8c0 tx=0x7f0e84009250 comp rx=0 tx=0).stop 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- 172.21.15.151:0/1171611628 shutdown_connections 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 --2- 172.21.15.151:0/1171611628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c13b1a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 --2- 172.21.15.151:0/1171611628 >> v2:172.21.15.172:3300/0 conn(0x7f0e8c0b0070 0x7f0e8c0aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- 172.21.15.151:0/1171611628 >> 172.21.15.151:0/1171611628 conn(0x7f0e8c070fe0 msgr2=0x7f0e8c0af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- 172.21.15.151:0/1171611628 shutdown_connections 2023-09-30T02:02:55.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- 172.21.15.151:0/1171611628 wait complete. 2023-09-30T02:02:55.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 Processor -- start 2023-09-30T02:02:55.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.430+0000 7f0e913bd700 1 -- start start 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e8c0aaec0 0x7f0e8c13e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 0x7f0e8c0a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0a5620 con 0x7f0e8c0b0440 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0a83a0 con 0x7f0e8c0aaec0 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0a5170 con 0x7f0e8c0b0070 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8a7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 0x7f0e8c0a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8a7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 0x7f0e8c0a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52414/0 (socket says 172.21.15.151:52414) 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8a7fc700 1 -- 172.21.15.151:0/801333870 learned_addr learned my addr 172.21.15.151:0/801333870 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53260/0 (socket says 172.21.15.151:53260) 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8affd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e8c0aaec0 0x7f0e8c13e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 msgr2=0x7f0e8c0a4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 0x7f0e8c0a4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e8c0aaec0 msgr2=0x7f0e8c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e8c0aaec0 0x7f0e8c13e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 -- 172.21.15.151:0/801333870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e840094f0 con 0x7f0e8c0b0440 2023-09-30T02:02:55.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e8b7fe700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c06e710 tx=0x7f0e84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400b860 con 0x7f0e8c0b0440 2023-09-30T02:02:55.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e84011e60 con 0x7f0e8c0b0440 2023-09-30T02:02:55.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c0a5390 con 0x7f0e8c0b0440 2023-09-30T02:02:55.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8401a860 con 0x7f0e8c0b0440 2023-09-30T02:02:55.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.434+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c0a6760 con 0x7f0e8c0b0440 2023-09-30T02:02:55.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c0670a0 con 0x7f0e8c0b0440 2023-09-30T02:02:55.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0e8401ac90 con 0x7f0e8c0b0440 2023-09-30T02:02:55.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e72ffd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 0x7f0e78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0e8407e050 con 0x7f0e8c0b0440 2023-09-30T02:02:55.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e8affd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 0x7f0e78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.438+0000 7f0e8affd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 0x7f0e78062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e80006f00 tx=0x7f0e80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.446+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e8404e8b0 con 0x7f0e8c0b0440 2023-09-30T02:02:55.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.638+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e8c0a5d60 con 0x7f0e8c0b0440 2023-09-30T02:02:55.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.638+0000 7f0e72ffd700 1 -- 172.21.15.151:0/801333870 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0e84015080 con 0x7f0e8c0b0440 2023-09-30T02:02:55.641 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:55.642 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:02:55.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 msgr2=0x7f0e78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 0x7f0e78062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e80006f00 tx=0x7f0e80008040 comp rx=0 tx=0).stop 2023-09-30T02:02:55.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 msgr2=0x7f0e8c0a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c06e710 tx=0x7f0e84017000 comp rx=0 tx=0).stop 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 shutdown_connections 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0e78060550 0x7f0e78062a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e8c0b0440 0x7f0e8c0a6000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e8c0b0070 0x7f0e8c0a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 --2- 172.21.15.151:0/801333870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e8c0aaec0 0x7f0e8c13e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 >> 172.21.15.151:0/801333870 conn(0x7f0e8c070fe0 msgr2=0x7f0e8c062d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:55.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 shutdown_connections 2023-09-30T02:02:55.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.642+0000 7f0e913bd700 1 -- 172.21.15.151:0/801333870 wait complete. 2023-09-30T02:02:55.655 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T02:02:55.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:02:55.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.754+0000 7f8984a18700 1 Processor -- start 2023-09-30T02:02:55.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.754+0000 7f8984a18700 1 -- start start 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8984a18700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f89800b0a80 0x7f898006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8984a18700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898006c3f0 con 0x7f898006ea20 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8984a18700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898006db10 con 0x7f8980139f20 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8984a18700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89800b0e50 con 0x7f89800b0a80 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8976ffd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f898006ea20 0x7f89800aa4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48036/0 (socket says 172.21.15.151:48036) 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f8976ffd700 1 -- 172.21.15.151:0/2606291778 learned_addr learned my addr 172.21.15.151:0/2606291778 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:55.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897f7fe700 1 --2- 172.21.15.151:0/2606291778 >> v2:172.21.15.172:3300/0 conn(0x7f89800b0a80 0x7f898006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897effd700 1 -- 172.21.15.151:0/2606291778 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010404917 0 0) 0x7f898006c3f0 con 0x7f898006ea20 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897effd700 1 -- 172.21.15.151:0/2606291778 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f896c003570 con 0x7f898006ea20 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897f7fe700 1 -- 172.21.15.151:0/2606291778 >> v1:172.21.15.172:6789/0 conn(0x7f8980139f20 legacy=0x7f8980144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897f7fe700 1 -- 172.21.15.151:0/2606291778 >> v1:172.21.15.151:6789/0 conn(0x7f898006ea20 legacy=0x7f89800aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897f7fe700 1 -- 172.21.15.151:0/2606291778 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898006dd80 con 0x7f89800b0a80 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897f7fe700 1 --2- 172.21.15.151:0/2606291778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89800b0a80 0x7f898006e4e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f89700098e0 tx=0x7f89700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be376f4159f77dbd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897effd700 1 -- 172.21.15.151:0/2606291778 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897000f6b0 con 0x7f89800b0a80 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897effd700 1 -- 172.21.15.151:0/2606291778 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897000f810 con 0x7f89800b0a80 2023-09-30T02:02:55.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.758+0000 7f897effd700 1 -- 172.21.15.151:0/2606291778 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897000fb90 con 0x7f89800b0a80 2023-09-30T02:02:55.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- 172.21.15.151:0/2606291778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89800b0a80 msgr2=0x7f898006e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 --2- 172.21.15.151:0/2606291778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89800b0a80 0x7f898006e4e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f89700098e0 tx=0x7f89700091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- 172.21.15.151:0/2606291778 shutdown_connections 2023-09-30T02:02:55.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 --2- 172.21.15.151:0/2606291778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89800b0a80 0x7f898006e4e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- 172.21.15.151:0/2606291778 >> 172.21.15.151:0/2606291778 conn(0x7f8980070510 msgr2=0x7f8980070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:55.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- 172.21.15.151:0/2606291778 shutdown_connections 2023-09-30T02:02:55.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- 172.21.15.151:0/2606291778 wait complete. 2023-09-30T02:02:55.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 Processor -- start 2023-09-30T02:02:55.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- start start 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f898006ea20 0x7f898013d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 0x7f89800a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800a5120 con 0x7f89800b0a80 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800a8370 con 0x7f898006ea20 2023-09-30T02:02:55.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8984a18700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898013e1b0 con 0x7f8980139f20 2023-09-30T02:02:55.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f8976ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 0x7f89800a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.762+0000 7f897ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f897ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52426/0 (socket says 172.21.15.151:52426) 2023-09-30T02:02:55.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f897ffff700 1 -- 172.21.15.151:0/201654709 learned_addr learned my addr 172.21.15.151:0/201654709 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:55.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f897f7fe700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f898006ea20 0x7f898013d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 -- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 msgr2=0x7f89800a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 -- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f898006ea20 msgr2=0x7f898013d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f898006ea20 0x7f898013d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 -- 172.21.15.151:0/201654709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970009420 con 0x7f89800b0a80 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f897ffff700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8976ffd700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 0x7f89800a4910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f896800c560 tx=0x7f8968007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968005b30 con 0x7f89800b0a80 2023-09-30T02:02:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8968005cb0 con 0x7f89800b0a80 2023-09-30T02:02:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89800a8cd0 con 0x7f89800b0a80 2023-09-30T02:02:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89800a91a0 con 0x7f89800b0a80 2023-09-30T02:02:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968007320 con 0x7f89800b0a80 2023-09-30T02:02:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89640050b0 con 0x7f89800b0a80 2023-09-30T02:02:55.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.766+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f896801b3b0 con 0x7f89800b0a80 2023-09-30T02:02:55.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.770+0000 7f8977fff700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 0x7f8960062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:55.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.770+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f896807b8d0 con 0x7f89800b0a80 2023-09-30T02:02:55.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.770+0000 7f897f7fe700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 0x7f8960062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:55.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.770+0000 7f897f7fe700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 0x7f8960062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f897000ffd0 tx=0x7f8970019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:55.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.782+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f896804c0b0 con 0x7f89800b0a80 2023-09-30T02:02:55.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.942+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8964005630 con 0x7f89800b0a80 2023-09-30T02:02:55.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.942+0000 7f8977fff700 1 -- 172.21.15.151:0/201654709 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13695 (secure 0 0 0) 0x7f896804f6d0 con 0x7f89800b0a80 2023-09-30T02:02:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:55.945 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":18,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:51.540012+0000","last_up_change":"2023-09-30T01:57:47.500496+0000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:57:47.499184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:57:48.763412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":31361},{"type":"v1","addr":"172.21.15.151:6818","nonce":31361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":31361},{"type":"v1","addr":"172.21.15.151:6824","nonce":31361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":31361},{"type":"v1","addr":"172.21.15.151:6831","nonce":31361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":31361},{"type":"v1","addr":"172.21.15.151:6828","nonce":31361}]},"public_addr":"172.21.15.151:6818/31361","cluster_addr":"172.21.15.151:6824/31361","heartbeat_back_addr":"172.21.15.151:6831/31361","heartbeat_front_addr":"172.21.15.151:6828/31361","state":["exists","up"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":31363},{"type":"v1","addr":"172.21.15.151:6811","nonce":31363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":31363},{"type":"v1","addr":"172.21.15.151:6815","nonce":31363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":31363},{"type":"v1","addr":"172.21.15.151:6823","nonce":31363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":31363},{"type":"v1","addr":"172.21.15.151:6819","nonce":31363}]},"public_addr":"172.21.15.151:6811/31363","cluster_addr":"172.21.15.151:6815/31363","heartbeat_back_addr":"172.21.15.151:6823/31363","heartbeat_front_addr":"172.21.15.151:6819/31363","state":["exists","up"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":31362},{"type":"v1","addr":"172.21.15.151:6810","nonce":31362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":31362},{"type":"v1","addr":"172.21.15.151:6816","nonce":31362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":31362},{"type":"v1","addr":"172.21.15.151:6829","nonce":31362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":31362},{"type":"v1","addr":"172.21.15.151:6825","nonce":31362}]},"public_addr":"172.21.15.151:6810/31362","cluster_addr":"172.21.15.151:6816/31362","heartbeat_back_addr":"172.21.15.151:6829/31362","heartbeat_front_addr":"172.21.15.151:6825/31362","state":["exists","up"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":31360},{"type":"v1","addr":"172.21.15.151:6801","nonce":31360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":31360},{"type":"v1","addr":"172.21.15.151:6803","nonce":31360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":31360},{"type":"v1","addr":"172.21.15.151:6807","nonce":31360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":31360},{"type":"v1","addr":"172.21.15.151:6805","nonce":31360}]},"public_addr":"172.21.15.151:6801/31360","cluster_addr":"172.21.15.151:6803/31360","heartbeat_back_addr":"172.21.15.151:6807/31360","heartbeat_front_addr":"172.21.15.151:6805/31360","state":["exists","up"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":31451},{"type":"v1","addr":"172.21.15.172:6801","nonce":31451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":31451},{"type":"v1","addr":"172.21.15.172:6803","nonce":31451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":31451},{"type":"v1","addr":"172.21.15.172:6807","nonce":31451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":31451},{"type":"v1","addr":"172.21.15.172:6805","nonce":31451}]},"public_addr":"172.21.15.172:6801/31451","cluster_addr":"172.21.15.172:6803/31451","heartbeat_back_addr":"172.21.15.172:6807/31451","heartbeat_front_addr":"172.21.15.172:6805/31451","state":["exists","up"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":31452},{"type":"v1","addr":"172.21.15.172:6811","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":31452},{"type":"v1","addr":"172.21.15.172:6814","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":31452},{"type":"v1","addr":"172.21.15.172:6820","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":31452},{"type":"v1","addr":"172.21.15.172:6817","nonce":31452}]},"public_addr":"172.21.15.172:6811/31452","cluster_addr":"172.21.15.172:6814/31452","heartbeat_back_addr":"172.21.15.172:6820/31452","heartbeat_front_addr":"172.21.15.172:6817/31452","state":["exists","up"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31453},{"type":"v1","addr":"172.21.15.172:6825","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":31453},{"type":"v1","addr":"172.21.15.172:6827","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":31453},{"type":"v1","addr":"172.21.15.172:6831","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":31453},{"type":"v1","addr":"172.21.15.172:6829","nonce":31453}]},"public_addr":"172.21.15.172:6825/31453","cluster_addr":"172.21.15.172:6827/31453","heartbeat_back_addr":"172.21.15.172:6831/31453","heartbeat_front_addr":"172.21.15.172:6829/31453","state":["exists","up"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":31450},{"type":"v1","addr":"172.21.15.172:6809","nonce":31450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":31450},{"type":"v1","addr":"172.21.15.172:6815","nonce":31450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":31450},{"type":"v1","addr":"172.21.15.172:6823","nonce":31450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":31450},{"type":"v1","addr":"172.21.15.172:6821","nonce":31450}]},"public_addr":"172.21.15.172:6809/31450","cluster_addr":"172.21.15.172:6815/31450","heartbeat_back_addr":"172.21.15.172:6823/31450","heartbeat_front_addr":"172.21.15.172:6821/31450","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.403867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.433944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.158817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.278191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.087404+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.063855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.494600+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.286608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 msgr2=0x7f8960062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 0x7f8960062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f897000ffd0 tx=0x7f8970019000 comp rx=0 tx=0).stop 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 msgr2=0x7f89800a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 0x7f89800a4910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f896800c560 tx=0x7f8968007d50 comp rx=0 tx=0).stop 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 shutdown_connections 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8960060550 0x7f8960062a00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8980139f20 0x7f89800a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89800b0a80 0x7f89800a4910 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 --2- 172.21.15.151:0/201654709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f898006ea20 0x7f898013d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 >> 172.21.15.151:0/201654709 conn(0x7f8980070510 msgr2=0x7f89800ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 shutdown_connections 2023-09-30T02:02:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:55.946+0000 7f8984a18700 1 -- 172.21.15.151:0/201654709 wait complete. 2023-09-30T02:02:55.959 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T02:02:55.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:02:56.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 Processor -- start 2023-09-30T02:02:56.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 -- start start 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f01280679f0 0x7f012806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012813a1f0 con 0x7f0128139680 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012806e960 con 0x7f01280679f0 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012dbec700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0128067dc0 con 0x7f0128136360 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f01280679f0 0x7f012806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012d3eb700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012d3eb700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52432/0 (socket says 172.21.15.151:52432) 2023-09-30T02:02:56.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f01280679f0 0x7f012806dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53274/0 (socket says 172.21.15.151:53274) 2023-09-30T02:02:56.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012d3eb700 1 -- 172.21.15.151:0/154244501 learned_addr learned my addr 172.21.15.151:0/154244501 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f01277fe700 1 -- 172.21.15.151:0/154244501 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6380429 0 0) 0x7f0118001110 con 0x7f0128139680 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f01277fe700 1 -- 172.21.15.151:0/154244501 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110003570 con 0x7f0128139680 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 -- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 msgr2=0x7f01281367e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 -- 172.21.15.151:0/154244501 >> v1:172.21.15.172:6789/0 conn(0x7f0128139680 legacy=0x7f0128139ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012cbea700 1 -- 172.21.15.151:0/154244501 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012806d8e0 con 0x7f01280679f0 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.058+0000 7f012d3eb700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012cbea700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01280679f0 0x7f012806dfe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f011c0098e0 tx=0x7f011c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e702f5a34ab6db9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f01277fe700 1 -- 172.21.15.151:0/154244501 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011630 con 0x7f01280679f0 2023-09-30T02:02:56.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f01277fe700 1 -- 172.21.15.151:0/154244501 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c011790 con 0x7f01280679f0 2023-09-30T02:02:56.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f01277fe700 1 -- 172.21.15.151:0/154244501 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011b50 con 0x7f01280679f0 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01280679f0 msgr2=0x7f012806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01280679f0 0x7f012806dfe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f011c0098e0 tx=0x7f011c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- 172.21.15.151:0/154244501 shutdown_connections 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0128136360 0x7f01281367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- 172.21.15.151:0/154244501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01280679f0 0x7f012806dfe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- 172.21.15.151:0/154244501 >> 172.21.15.151:0/154244501 conn(0x7f01280b03f0 msgr2=0x7f01280b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- 172.21.15.151:0/154244501 shutdown_connections 2023-09-30T02:02:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- 172.21.15.151:0/154244501 wait complete. 2023-09-30T02:02:56.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 Processor -- start 2023-09-30T02:02:56.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 -- start start 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0128136360 0x7f012813cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.062+0000 7f012dbec700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0128139680 0x7f0128134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128134880 con 0x7f0128139680 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012813aca0 con 0x7f0128136360 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012813b870 con 0x7f01280679f0 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52434/0 (socket says 172.21.15.151:52434) 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 -- 172.21.15.151:0/684755019 learned_addr learned my addr 172.21.15.151:0/684755019 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012d3eb700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0128139680 0x7f0128134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f0127fff700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0128136360 0x7f012813cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 -- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0128136360 msgr2=0x7f012813cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0128136360 0x7f012813cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 -- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0128139680 msgr2=0x7f0128134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0128139680 0x7f0128134cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 -- 172.21.15.151:0/684755019 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012000abf0 con 0x7f01280679f0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012cbea700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f01280adb50 tx=0x7f011c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011b10 con 0x7f01280679f0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c026450 con 0x7f01280679f0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c009420 con 0x7f01280679f0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0267f0 con 0x7f01280679f0 2023-09-30T02:02:56.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01281353b0 con 0x7f01280679f0 2023-09-30T02:02:56.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128062d60 con 0x7f01280679f0 2023-09-30T02:02:56.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.066+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f011c02f3f0 con 0x7f01280679f0 2023-09-30T02:02:56.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.070+0000 7f0124ff9700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 0x7f0114062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.070+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f011c07ddd0 con 0x7f01280679f0 2023-09-30T02:02:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.070+0000 7f0127fff700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 0x7f0114062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.070+0000 7f0127fff700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 0x7f0114062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f012806c7e0 tx=0x7f0118007bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.074+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f011c04e710 con 0x7f01280679f0 2023-09-30T02:02:56.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.234+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0128134680 con 0x7f0114060550 2023-09-30T02:02:56.243 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f0124ff9700 1 -- 172.21.15.151:0/684755019 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70509 (secure 0 0 0) 0x7f0128134680 con 0x7f0114060550 2023-09-30T02:02:56.244 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:56.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 msgr2=0x7f0114062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 0x7f0114062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f012806c7e0 tx=0x7f0118007bc0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 msgr2=0x7f012813c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f01280adb50 tx=0x7f011c017000 comp rx=0 tx=0).stop 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 shutdown_connections 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0114060550 0x7f0114062a00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0128139680 0x7f0128134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0128136360 0x7f012813cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 --2- 172.21.15.151:0/684755019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01280679f0 0x7f012813c5d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.242+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 >> 172.21.15.151:0/684755019 conn(0x7f01280b03f0 msgr2=0x7f012806b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.246+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 shutdown_connections 2023-09-30T02:02:56.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.246+0000 7f012dbec700 1 -- 172.21.15.151:0/684755019 wait complete. 2023-09-30T02:02:56.248 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T02:02:56.260 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2023-09-30T02:02:55.528116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":43136,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":40640,"kb_avail":754931584,"statfs":{"total":773094113280,"available":773049942016,"internally_reserved":0,"allocated":2490368,"data_stored":1519970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41615360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.544219+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:50.544219+0000","last_peered":"2023-09-30T01:57:50.544219+0000","last_clean":"2023-09-30T01:57:50.544219+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:50.544219+0000","last_undegraded":"2023-09-30T01:57:50.544219+0000","last_fullsized":"2023-09-30T01:57:50.544219+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:18:26.843921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111495+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.111495+0000","last_peered":"2023-09-30T01:57:51.111495+0000","last_clean":"2023-09-30T01:57:51.111495+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.111495+0000","last_undegraded":"2023-09-30T01:57:51.111495+0000","last_fullsized":"2023-09-30T01:57:51.111495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111284+0000","last_change":"2023-09-30T01:57:51.106990+0000","last_active":"2023-09-30T01:57:51.111284+0000","last_peered":"2023-09-30T01:57:51.111284+0000","last_clean":"2023-09-30T01:57:51.111284+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T01:57:51.111284+0000","last_undegraded":"2023-09-30T01:57:51.111284+0000","last_fullsized":"2023-09-30T01:57:51.111284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:40.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.551200+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.551200+0000","last_peered":"2023-09-30T01:57:50.551200+0000","last_clean":"2023-09-30T01:57:50.551200+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.551200+0000","last_undegraded":"2023-09-30T01:57:50.551200+0000","last_fullsized":"2023-09-30T01:57:50.551200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.702970+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.702970+0000","last_peered":"2023-09-30T01:57:50.702970+0000","last_clean":"2023-09-30T01:57:50.702970+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.702970+0000","last_undegraded":"2023-09-30T01:57:50.702970+0000","last_fullsized":"2023-09-30T01:57:50.702970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549262+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.549262+0000","last_peered":"2023-09-30T01:57:50.549262+0000","last_clean":"2023-09-30T01:57:50.549262+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.549262+0000","last_undegraded":"2023-09-30T01:57:50.549262+0000","last_fullsized":"2023-09-30T01:57:50.549262+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553788+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.553788+0000","last_peered":"2023-09-30T01:57:50.553788+0000","last_clean":"2023-09-30T01:57:50.553788+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.553788+0000","last_undegraded":"2023-09-30T01:57:50.553788+0000","last_fullsized":"2023-09-30T01:57:50.553788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.129346+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.129346+0000","last_peered":"2023-09-30T01:57:51.129346+0000","last_clean":"2023-09-30T01:57:51.129346+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.129346+0000","last_undegraded":"2023-09-30T01:57:51.129346+0000","last_fullsized":"2023-09-30T01:57:51.129346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.60299999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.0700000000000001,"5min":1.258,"15min":1.258},"last":1.002},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.97699999999999998,"15min":1.01},"min":{"1min":0.64800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2909999999999999,"5min":1.353,"15min":1.353},"last":0.78300000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.64900000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2829999999999999,"5min":1.365,"15min":1.365},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89900000000000002,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.3109999999999999,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.034}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96399999999999997,"15min":0.92800000000000005},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95299999999999996,"15min":0.97599999999999998},"min":{"1min":0.68000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.2050000000000001,"5min":1.466,"15min":1.466},"last":0.83999999999999997}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.997,"15min":1.0149999999999999},"min":{"1min":0.66600000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.421,"5min":1.446,"15min":1.446},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.0940000000000001,"15min":1.0860000000000001},"min":{"1min":0.878,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.3959999999999999,"5min":1.403,"15min":1.403},"last":1.087}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.92200000000000004,"15min":0.92000000000000004},"min":{"1min":0.67400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0920000000000001,"15min":1.0940000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.456,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.1240000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0429999999999999,"15min":1.052},"min":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.3520000000000001,"5min":1.55,"15min":1.55},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0880000000000001,"15min":1.097},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.93100000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1739999999999999,"15min":1.196},"min":{"1min":0.86299999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.631,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2050000000000001,"15min":1.2250000000000001},"min":{"1min":0.95399999999999996,"5min":0.753,"15min":0.753},"max":{"1min":1.4630000000000001,"5min":1.627,"15min":1.627},"last":0.91400000000000003}]}]},{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91200000000000003,"15min":0.89800000000000002},"min":{"1min":0.58999999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.123},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89100000000000001,"15min":0.91100000000000003},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.141,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.72199999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95599999999999996,"15min":0.91600000000000004},"min":{"1min":0.66700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0409999999999999,"15min":1.0629999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.1479999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91200000000000003},"min":{"1min":0.68799999999999994,"5min":0.63,"15min":0.63},"max":{"1min":1.248,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.66900000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.153,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.999}]},{"osd":3,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95699999999999996,"15min":0.96799999999999997},"min":{"1min":0.68899999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1759999999999999,"5min":1.218,"15min":1.218},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0109999999999999,"15min":1.0209999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.90800000000000003}]},{"osd":4,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90700000000000003,"15min":0.94299999999999995},"min":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2070000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.0669999999999999,"15min":1.101},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.405,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0409999999999999,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4910000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.153},"min":{"1min":0.89600000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.2869999999999999,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.1870000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.119,"15min":1.115},"min":{"1min":0.89600000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.3640000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.159,"15min":1.1819999999999999},"min":{"1min":0.94699999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.419,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.2569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83799999999999997,"15min":0.83599999999999997},"min":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.98099999999999998,"15min":1.026},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85999999999999999,"15min":0.85599999999999998},"min":{"1min":0.66100000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.262,"15min":1.262},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.92800000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.97899999999999998,"15min":1.028},"min":{"1min":0.66300000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.214,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.876}]},{"osd":3,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.242,"5min":1.575,"15min":1.575},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.95099999999999996},"min":{"1min":0.66900000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.1240000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90700000000000003,"15min":0.89400000000000002},"min":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.028,"5min":1.04,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.216,"5min":1.329,"15min":1.329},"last":1.2010000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1950000000000001,"5min":1.669,"15min":1.669},"last":1.012},{"interface":"front","average":{"1min":1.103,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.2949999999999999,"5min":1.75,"15min":1.75},"last":1.075}]},{"osd":7,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":1.23},"min":{"1min":1.03,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.349,"5min":1.859,"15min":1.859},"last":1.171},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.034,"15min":1.032},"min":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3799999999999999,"5min":1.526,"15min":1.526},"last":1.25}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":0.94099999999999995},"min":{"1min":0.67500000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1970000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1890000000000001,"5min":1.327,"15min":1.327},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.131,"15min":1.167},"min":{"1min":0.96499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.4410000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.1559999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.95699999999999996,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.163,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.73299999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.78600000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.3500000000000001,"5min":1.472,"15min":1.472},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.997,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.4510000000000001,"5min":3.9889999999999999,"15min":3.9889999999999999},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.89900000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.996,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.113},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4099999999999999,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.23}]},{"osd":6,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1020000000000001,"15min":1.048},"min":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.85999999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.2549999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.1819999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1319999999999999,"15min":1.129},"min":{"1min":0.96699999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.5549999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.1519999999999999,"15min":1.1679999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3100000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91200000000000003,"5min":2.29,"15min":2.29},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1659999999999999,"5min":1.569,"15min":1.569},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":1.28,"15min":1.28},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84899999999999998,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.499,"15min":0.499},"max":{"1min":1.3600000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.745}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.0880000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1200000000000001,"5min":1.363,"15min":1.363},"last":1.0249999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0700000000000001,"15min":1.1459999999999999},"min":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.351,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0720000000000001,"15min":1.0840000000000001},"min":{"1min":0.85799999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.532,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.98099999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.046,"15min":1.0800000000000001},"min":{"1min":0.83199999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.474,"5min":1.645,"15min":1.645},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.80800000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.371,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.879}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.2050000000000001,"15min":1.2749999999999999},"min":{"1min":0.79700000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.704,"5min":2.516,"15min":2.516},"last":1.149},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.206,"15min":1.2509999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6679999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":1.002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.524,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3140000000000001},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.4370000000000001,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.1120000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5168,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366672,"statfs":{"total":96636764160,"available":96631472128,"internally_reserved":0,"allocated":172032,"data_stored":56052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65600000000000003,"15min":0.65300000000000002},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.034,"5min":1.181,"15min":1.181},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.73399999999999999,"15min":0.754},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96799999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.86799999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.85899999999999999},"min":{"1min":0.59099999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.401,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.042,"15min":1.0840000000000001},"min":{"1min":0.72499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5580000000000001,"5min":1.819,"15min":1.819},"last":1.091}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0669999999999999,"15min":1.089},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0860000000000001,"15min":1.1060000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.653,"5min":1.659,"15min":1.659},"last":1.002}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.163},"min":{"1min":0.79600000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.667,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.149,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.97999999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1459999999999999,"15min":1.1539999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.224,"15min":1.23},"min":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.0629999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.0549999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.76600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97899999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86599999999999999,"5min":1.296,"15min":1.296},"last":0.624},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.87},"min":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1639999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82399999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.874,"15min":0.88500000000000001},"min":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.337,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0449999999999999,"5min":1.218,"15min":1.218},"last":1.04}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97099999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.292,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.093,"5min":1.0920000000000001,"15min":1.119},"min":{"1min":0.79300000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.88800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4910000000000001,"5min":1.625,"15min":1.625},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.135,"15min":1.171},"min":{"1min":0.89400000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.96899999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1539999999999999,"15min":1.2},"min":{"1min":0.86599999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.091,"5min":1.153,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.351,"5min":1.778,"15min":1.778},"last":1.242}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2370000000000001},"min":{"1min":0.95599999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.113},{"interface":"front","average":{"1min":1.151,"5min":1.2170000000000001,"15min":1.262},"min":{"1min":0.96599999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.421,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1819999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":585928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66100000000000003,"15min":0.64600000000000002},"min":{"1min":0.51900000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90900000000000003,"5min":1.006,"15min":1.006},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.76200000000000001,"15min":0.752},"min":{"1min":0.622,"5min":0.441,"15min":0.441},"max":{"1min":1.0349999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.754}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90300000000000002,"5min":1.087,"15min":1.087},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.54500000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.97099999999999997,"5min":1.161,"15min":1.161},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.878},"min":{"1min":0.628,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.056,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0369999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.331,"5min":1.448,"15min":1.448},"last":1.026}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.002,"15min":1.052},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.127,"5min":1.468,"15min":1.468},"last":1.081},{"interface":"front","average":{"1min":1.077,"5min":1.0840000000000001,"15min":1.1160000000000001},"min":{"1min":0.86199999999999999,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4970000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.165}]},{"osd":6,"last update":"Sat Sep 30 02:02:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0920000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1899999999999999,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.147},"min":{"1min":0.81599999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.5609999999999999,"5min":1.768,"15min":1.768},"last":1.0469999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.155,"15min":1.167},"min":{"1min":0.94799999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.1739999999999999,"15min":1.2090000000000001},"min":{"1min":0.82799999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.296,"5min":1.669,"15min":1.669},"last":1.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:02:56.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:02:56.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 Processor -- start 2023-09-30T02:02:56.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 -- start start 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5e100b44f0 0x7f5e1006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006ea60 0x7f5e1006eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e100adf40 con 0x7f5e1006b770 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1006b590 con 0x7f5e100b44f0 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e16be8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1006d020 con 0x7f5e1006ea60 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5e100b44f0 0x7f5e1006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5e100b44f0 0x7f5e1006b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53304/0 (socket says 172.21.15.151:53304) 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 -- 172.21.15.151:0/1997396703 learned_addr learned my addr 172.21.15.151:0/1997396703 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e163e7700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006ea60 0x7f5e1006eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 -- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006ea60 msgr2=0x7f5e1006eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:56.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006ea60 0x7f5e1006eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 -- 172.21.15.151:0/1997396703 >> v1:172.21.15.172:6789/0 conn(0x7f5e1006b770 legacy=0x7f5e10140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 -- 172.21.15.151:0/1997396703 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e100b48c0 con 0x7f5e100b44f0 2023-09-30T02:02:56.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e15be6700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e100b44f0 0x7f5e1006b050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009f20 tx=0x7f5e08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc6279ba18fcb7b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e14be4700 1 -- 172.21.15.151:0/1997396703 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800e070 con 0x7f5e100b44f0 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e14be4700 1 -- 172.21.15.151:0/1997396703 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800b9d0 con 0x7f5e100b44f0 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.370+0000 7f5e14be4700 1 -- 172.21.15.151:0/1997396703 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080092e0 con 0x7f5e100b44f0 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e100b44f0 msgr2=0x7f5e1006b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e100b44f0 0x7f5e1006b050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009f20 tx=0x7f5e08009840 comp rx=0 tx=0).stop 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- 172.21.15.151:0/1997396703 shutdown_connections 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006ea60 0x7f5e1006eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1997396703 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e100b44f0 0x7f5e1006b050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- 172.21.15.151:0/1997396703 >> 172.21.15.151:0/1997396703 conn(0x7f5e10070fe0 msgr2=0x7f5e100b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- 172.21.15.151:0/1997396703 shutdown_connections 2023-09-30T02:02:56.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- 172.21.15.151:0/1997396703 wait complete. 2023-09-30T02:02:56.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 Processor -- start 2023-09-30T02:02:56.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- start start 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e1006ea60 0x7f5e100a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 0x7f5e100a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100a91b0 con 0x7f5e1006ea60 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100aae40 con 0x7f5e100b44f0 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e16be8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1013ba30 con 0x7f5e1006b770 2023-09-30T02:02:56.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e15be6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e15be6700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52456/0 (socket says 172.21.15.151:52456) 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e15be6700 1 -- 172.21.15.151:0/1692174518 learned_addr learned my addr 172.21.15.151:0/1692174518 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e153e5700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e1006ea60 0x7f5e100a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.374+0000 7f5e163e7700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 0x7f5e100a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 -- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 msgr2=0x7f5e100a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 0x7f5e100a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 -- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e1006ea60 msgr2=0x7f5e100a8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e1006ea60 0x7f5e100a8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 -- 172.21.15.151:0/1692174518 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08009ae0 con 0x7f5e1006b770 2023-09-30T02:02:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e163e7700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 0x7f5e100a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:56.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e15be6700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003440 tx=0x7f5e08018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080089d0 con 0x7f5e1006b770 2023-09-30T02:02:56.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08008b30 con 0x7f5e1006b770 2023-09-30T02:02:56.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1013bbf0 con 0x7f5e1006b770 2023-09-30T02:02:56.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0801c410 con 0x7f5e1006b770 2023-09-30T02:02:56.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e10151920 con 0x7f5e1006b770 2023-09-30T02:02:56.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e100008d0 con 0x7f5e1006b770 2023-09-30T02:02:56.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5e08003040 con 0x7f5e1006b770 2023-09-30T02:02:56.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 0x7f5df4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5e0807e240 con 0x7f5e1006b770 2023-09-30T02:02:56.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.378+0000 7f5e153e5700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 0x7f5df4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.382+0000 7f5e153e5700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 0x7f5df4062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5e100ab2c0 tx=0x7f5e04007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.386+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5e0804ea80 con 0x7f5e1006b770 2023-09-30T02:02:56.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.538+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e10151920 con 0x7f5df4060550 2023-09-30T02:02:56.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.546+0000 7f5e01ffb700 1 -- 172.21.15.151:0/1692174518 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70509 (secure 0 0 0) 0x7f5e10151920 con 0x7f5df4060550 2023-09-30T02:02:56.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:56.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 msgr2=0x7f5df4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 0x7f5df4062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5e100ab2c0 tx=0x7f5e04007ec0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 msgr2=0x7f5e10150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003440 tx=0x7f5e08018000 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 shutdown_connections 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5df4060550 0x7f5df4062a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e100b44f0 0x7f5e100a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e1006ea60 0x7f5e100a8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 --2- 172.21.15.151:0/1692174518 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e1006b770 0x7f5e10150c30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 >> 172.21.15.151:0/1692174518 conn(0x7f5e10070fe0 msgr2=0x7f5e100aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 shutdown_connections 2023-09-30T02:02:56.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.550+0000 7f5e16be8700 1 -- 172.21.15.151:0/1692174518 wait complete. 2023-09-30T02:02:56.553 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T02:02:56.565 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2023-09-30T02:02:55.528116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":43136,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":40640,"kb_avail":754931584,"statfs":{"total":773094113280,"available":773049942016,"internally_reserved":0,"allocated":2490368,"data_stored":1519970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41615360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.544219+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:50.544219+0000","last_peered":"2023-09-30T01:57:50.544219+0000","last_clean":"2023-09-30T01:57:50.544219+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:50.544219+0000","last_undegraded":"2023-09-30T01:57:50.544219+0000","last_fullsized":"2023-09-30T01:57:50.544219+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:18:26.843921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111495+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.111495+0000","last_peered":"2023-09-30T01:57:51.111495+0000","last_clean":"2023-09-30T01:57:51.111495+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.111495+0000","last_undegraded":"2023-09-30T01:57:51.111495+0000","last_fullsized":"2023-09-30T01:57:51.111495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111284+0000","last_change":"2023-09-30T01:57:51.106990+0000","last_active":"2023-09-30T01:57:51.111284+0000","last_peered":"2023-09-30T01:57:51.111284+0000","last_clean":"2023-09-30T01:57:51.111284+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T01:57:51.111284+0000","last_undegraded":"2023-09-30T01:57:51.111284+0000","last_fullsized":"2023-09-30T01:57:51.111284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:40.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.551200+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.551200+0000","last_peered":"2023-09-30T01:57:50.551200+0000","last_clean":"2023-09-30T01:57:50.551200+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.551200+0000","last_undegraded":"2023-09-30T01:57:50.551200+0000","last_fullsized":"2023-09-30T01:57:50.551200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.702970+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.702970+0000","last_peered":"2023-09-30T01:57:50.702970+0000","last_clean":"2023-09-30T01:57:50.702970+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.702970+0000","last_undegraded":"2023-09-30T01:57:50.702970+0000","last_fullsized":"2023-09-30T01:57:50.702970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549262+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.549262+0000","last_peered":"2023-09-30T01:57:50.549262+0000","last_clean":"2023-09-30T01:57:50.549262+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.549262+0000","last_undegraded":"2023-09-30T01:57:50.549262+0000","last_fullsized":"2023-09-30T01:57:50.549262+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553788+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.553788+0000","last_peered":"2023-09-30T01:57:50.553788+0000","last_clean":"2023-09-30T01:57:50.553788+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.553788+0000","last_undegraded":"2023-09-30T01:57:50.553788+0000","last_fullsized":"2023-09-30T01:57:50.553788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.129346+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.129346+0000","last_peered":"2023-09-30T01:57:51.129346+0000","last_clean":"2023-09-30T01:57:51.129346+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.129346+0000","last_undegraded":"2023-09-30T01:57:51.129346+0000","last_fullsized":"2023-09-30T01:57:51.129346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.60299999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.0700000000000001,"5min":1.258,"15min":1.258},"last":1.002},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.97699999999999998,"15min":1.01},"min":{"1min":0.64800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2909999999999999,"5min":1.353,"15min":1.353},"last":0.78300000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.64900000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2829999999999999,"5min":1.365,"15min":1.365},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89900000000000002,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.3109999999999999,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.034}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96399999999999997,"15min":0.92800000000000005},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95299999999999996,"15min":0.97599999999999998},"min":{"1min":0.68000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.2050000000000001,"5min":1.466,"15min":1.466},"last":0.83999999999999997}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.997,"15min":1.0149999999999999},"min":{"1min":0.66600000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.421,"5min":1.446,"15min":1.446},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.0940000000000001,"15min":1.0860000000000001},"min":{"1min":0.878,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.3959999999999999,"5min":1.403,"15min":1.403},"last":1.087}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.92200000000000004,"15min":0.92000000000000004},"min":{"1min":0.67400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0920000000000001,"15min":1.0940000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.456,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.1240000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0429999999999999,"15min":1.052},"min":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.3520000000000001,"5min":1.55,"15min":1.55},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0880000000000001,"15min":1.097},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.93100000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1739999999999999,"15min":1.196},"min":{"1min":0.86299999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.631,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2050000000000001,"15min":1.2250000000000001},"min":{"1min":0.95399999999999996,"5min":0.753,"15min":0.753},"max":{"1min":1.4630000000000001,"5min":1.627,"15min":1.627},"last":0.91400000000000003}]}]},{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91200000000000003,"15min":0.89800000000000002},"min":{"1min":0.58999999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.123},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89100000000000001,"15min":0.91100000000000003},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.141,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.72199999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95599999999999996,"15min":0.91600000000000004},"min":{"1min":0.66700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0409999999999999,"15min":1.0629999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.1479999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91200000000000003},"min":{"1min":0.68799999999999994,"5min":0.63,"15min":0.63},"max":{"1min":1.248,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.66900000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.153,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.999}]},{"osd":3,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95699999999999996,"15min":0.96799999999999997},"min":{"1min":0.68899999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1759999999999999,"5min":1.218,"15min":1.218},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0109999999999999,"15min":1.0209999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.90800000000000003}]},{"osd":4,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90700000000000003,"15min":0.94299999999999995},"min":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2070000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.0669999999999999,"15min":1.101},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.405,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0409999999999999,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4910000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.153},"min":{"1min":0.89600000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.2869999999999999,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.1870000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.119,"15min":1.115},"min":{"1min":0.89600000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.3640000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.159,"15min":1.1819999999999999},"min":{"1min":0.94699999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.419,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.2569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83799999999999997,"15min":0.83599999999999997},"min":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.98099999999999998,"15min":1.026},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85999999999999999,"15min":0.85599999999999998},"min":{"1min":0.66100000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.262,"15min":1.262},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.92800000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.97899999999999998,"15min":1.028},"min":{"1min":0.66300000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.214,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.876}]},{"osd":3,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.242,"5min":1.575,"15min":1.575},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.95099999999999996},"min":{"1min":0.66900000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.1240000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90700000000000003,"15min":0.89400000000000002},"min":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.028,"5min":1.04,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.216,"5min":1.329,"15min":1.329},"last":1.2010000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1950000000000001,"5min":1.669,"15min":1.669},"last":1.012},{"interface":"front","average":{"1min":1.103,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.2949999999999999,"5min":1.75,"15min":1.75},"last":1.075}]},{"osd":7,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":1.23},"min":{"1min":1.03,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.349,"5min":1.859,"15min":1.859},"last":1.171},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.034,"15min":1.032},"min":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3799999999999999,"5min":1.526,"15min":1.526},"last":1.25}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":0.94099999999999995},"min":{"1min":0.67500000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1970000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1890000000000001,"5min":1.327,"15min":1.327},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.131,"15min":1.167},"min":{"1min":0.96499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.4410000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.1559999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.95699999999999996,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.163,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.73299999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.78600000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.3500000000000001,"5min":1.472,"15min":1.472},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.997,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.4510000000000001,"5min":3.9889999999999999,"15min":3.9889999999999999},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.89900000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.996,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.113},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4099999999999999,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.23}]},{"osd":6,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1020000000000001,"15min":1.048},"min":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.85999999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.2549999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.1819999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1319999999999999,"15min":1.129},"min":{"1min":0.96699999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.5549999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.1519999999999999,"15min":1.1679999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3100000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91200000000000003,"5min":2.29,"15min":2.29},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1659999999999999,"5min":1.569,"15min":1.569},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":1.28,"15min":1.28},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84899999999999998,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.499,"15min":0.499},"max":{"1min":1.3600000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.745}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.0880000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1200000000000001,"5min":1.363,"15min":1.363},"last":1.0249999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0700000000000001,"15min":1.1459999999999999},"min":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.351,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0720000000000001,"15min":1.0840000000000001},"min":{"1min":0.85799999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.532,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.98099999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.046,"15min":1.0800000000000001},"min":{"1min":0.83199999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.474,"5min":1.645,"15min":1.645},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.80800000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.371,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.879}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.2050000000000001,"15min":1.2749999999999999},"min":{"1min":0.79700000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.704,"5min":2.516,"15min":2.516},"last":1.149},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.206,"15min":1.2509999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6679999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":1.002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.524,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3140000000000001},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.4370000000000001,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.1120000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5168,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366672,"statfs":{"total":96636764160,"available":96631472128,"internally_reserved":0,"allocated":172032,"data_stored":56052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65600000000000003,"15min":0.65300000000000002},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.034,"5min":1.181,"15min":1.181},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.73399999999999999,"15min":0.754},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96799999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.86799999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.85899999999999999},"min":{"1min":0.59099999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.401,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.042,"15min":1.0840000000000001},"min":{"1min":0.72499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5580000000000001,"5min":1.819,"15min":1.819},"last":1.091}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0669999999999999,"15min":1.089},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0860000000000001,"15min":1.1060000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.653,"5min":1.659,"15min":1.659},"last":1.002}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.163},"min":{"1min":0.79600000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.667,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.149,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.97999999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1459999999999999,"15min":1.1539999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.224,"15min":1.23},"min":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.0629999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.0549999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.76600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97899999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86599999999999999,"5min":1.296,"15min":1.296},"last":0.624},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.87},"min":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1639999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82399999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.874,"15min":0.88500000000000001},"min":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.337,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0449999999999999,"5min":1.218,"15min":1.218},"last":1.04}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97099999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.292,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.093,"5min":1.0920000000000001,"15min":1.119},"min":{"1min":0.79300000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.88800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4910000000000001,"5min":1.625,"15min":1.625},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.135,"15min":1.171},"min":{"1min":0.89400000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.96899999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1539999999999999,"15min":1.2},"min":{"1min":0.86599999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.091,"5min":1.153,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.351,"5min":1.778,"15min":1.778},"last":1.242}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2370000000000001},"min":{"1min":0.95599999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.113},{"interface":"front","average":{"1min":1.151,"5min":1.2170000000000001,"15min":1.262},"min":{"1min":0.96599999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.421,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1819999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":585928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66100000000000003,"15min":0.64600000000000002},"min":{"1min":0.51900000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90900000000000003,"5min":1.006,"15min":1.006},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.76200000000000001,"15min":0.752},"min":{"1min":0.622,"5min":0.441,"15min":0.441},"max":{"1min":1.0349999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.754}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90300000000000002,"5min":1.087,"15min":1.087},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.54500000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.97099999999999997,"5min":1.161,"15min":1.161},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.878},"min":{"1min":0.628,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.056,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0369999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.331,"5min":1.448,"15min":1.448},"last":1.026}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.002,"15min":1.052},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.127,"5min":1.468,"15min":1.468},"last":1.081},{"interface":"front","average":{"1min":1.077,"5min":1.0840000000000001,"15min":1.1160000000000001},"min":{"1min":0.86199999999999999,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4970000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.165}]},{"osd":6,"last update":"Sat Sep 30 02:02:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0920000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1899999999999999,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.147},"min":{"1min":0.81599999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.5609999999999999,"5min":1.768,"15min":1.768},"last":1.0469999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.155,"15min":1.167},"min":{"1min":0.94799999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.1739999999999999,"15min":1.2090000000000001},"min":{"1min":0.82799999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.296,"5min":1.669,"15min":1.669},"last":1.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:02:56.567 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T02:02:56.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:02:56.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 Processor -- start 2023-09-30T02:02:56.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 -- start start 2023-09-30T02:02:56.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5b2806f3e0 0x7f5b2813d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2806ea00 con 0x7f5b280aa9b0 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b28067ea0 con 0x7f5b2806f3e0 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b27fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2806d8f0 con 0x7f5b28067ad0 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b267fc700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5b2806f3e0 0x7f5b2813d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b267fc700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5b2806f3e0 0x7f5b2813d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53316/0 (socket says 172.21.15.151:53316) 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52472/0 (socket says 172.21.15.151:52472) 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b267fc700 1 -- 172.21.15.151:0/3623050082 learned_addr learned my addr 172.21.15.151:0/3623050082 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.670+0000 7f5b277fe700 1 --1- 172.21.15.151:0/3623050082 >> v1:172.21.15.172:6789/0 conn(0x7f5b280aa9b0 0x7f5b280aae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:58872/0 (socket says 172.21.15.151:58872) 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b25ffb700 1 -- 172.21.15.151:0/3623050082 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807892013 0 0) 0x7f5b2806ea00 con 0x7f5b280aa9b0 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 -- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 msgr2=0x7f5b2813cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 -- 172.21.15.151:0/3623050082 >> v1:172.21.15.172:6789/0 conn(0x7f5b280aa9b0 legacy=0x7f5b280aae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 -- 172.21.15.151:0/3623050082 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280ac840 con 0x7f5b2806f3e0 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b26ffd700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b2806f3e0 0x7f5b2813d1c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00ab50 tx=0x7f5b1c00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72620f273e466a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b25ffb700 1 -- 172.21.15.151:0/3623050082 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00ca40 con 0x7f5b2806f3e0 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b25ffb700 1 -- 172.21.15.151:0/3623050082 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c00cba0 con 0x7f5b2806f3e0 2023-09-30T02:02:56.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b25ffb700 1 -- 172.21.15.151:0/3623050082 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c018400 con 0x7f5b2806f3e0 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b2806f3e0 msgr2=0x7f5b2813d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b2806f3e0 0x7f5b2813d1c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00ab50 tx=0x7f5b1c00a3f0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- 172.21.15.151:0/3623050082 shutdown_connections 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b2806f3e0 0x7f5b2813d1c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- 172.21.15.151:0/3623050082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b28067ad0 0x7f5b2813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- 172.21.15.151:0/3623050082 >> 172.21.15.151:0/3623050082 conn(0x7f5b280b03e0 msgr2=0x7f5b280b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- 172.21.15.151:0/3623050082 shutdown_connections 2023-09-30T02:02:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- 172.21.15.151:0/3623050082 wait complete. 2023-09-30T02:02:56.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 Processor -- start 2023-09-30T02:02:56.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- start start 2023-09-30T02:02:56.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 0x7f5b28139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b280aa9b0 0x7f5b2813a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280a8850 con 0x7f5b28067ad0 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2813c670 con 0x7f5b280aa9b0 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b27fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2813c320 con 0x7f5b2806f3e0 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b267fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 0x7f5b28139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53324/0 (socket says 172.21.15.151:53324) 2023-09-30T02:02:56.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b26ffd700 1 -- 172.21.15.151:0/2093641650 learned_addr learned my addr 172.21.15.151:0/2093641650 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.674+0000 7f5b277fe700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b280aa9b0 0x7f5b2813a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 -- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 msgr2=0x7f5b28139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 0x7f5b28139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 -- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b280aa9b0 msgr2=0x7f5b2813a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b280aa9b0 0x7f5b2813a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 -- 172.21.15.151:0/2093641650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009420 con 0x7f5b28067ad0 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b267fc700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 0x7f5b28139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b26ffd700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800ec80 tx=0x7f5b180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800c850 con 0x7f5b28067ad0 2023-09-30T02:02:56.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800c9b0 con 0x7f5b28067ad0 2023-09-30T02:02:56.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c00a690 con 0x7f5b28067ad0 2023-09-30T02:02:56.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800cd30 con 0x7f5b28067ad0 2023-09-30T02:02:56.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28138880 con 0x7f5b28067ad0 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b280008d0 con 0x7f5b28067ad0 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5b18013680 con 0x7f5b28067ad0 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 0x7f5b14062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5b1807c910 con 0x7f5b28067ad0 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.678+0000 7f5b267fc700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 0x7f5b14062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.682+0000 7f5b267fc700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 0x7f5b14062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00ab50 tx=0x7f5b1c01a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.686+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5b18050b80 con 0x7f5b28067ad0 2023-09-30T02:02:56.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.842+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b2813c4e0 con 0x7f5b14060550 2023-09-30T02:02:56.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.846+0000 7f5b0effd700 1 -- 172.21.15.151:0/2093641650 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70509 (secure 0 0 0) 0x7f5b2813c4e0 con 0x7f5b14060550 2023-09-30T02:02:56.850 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:56.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 msgr2=0x7f5b14062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 0x7f5b14062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00ab50 tx=0x7f5b1c01a000 comp rx=0 tx=0).stop 2023-09-30T02:02:56.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 msgr2=0x7f5b28139660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800ec80 tx=0x7f5b180091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 shutdown_connections 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5b14060550 0x7f5b14062a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b280aa9b0 0x7f5b2813a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b2806f3e0 0x7f5b28139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 --2- 172.21.15.151:0/2093641650 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b28067ad0 0x7f5b28139660 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 >> 172.21.15.151:0/2093641650 conn(0x7f5b280b03e0 msgr2=0x7f5b281337a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 shutdown_connections 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.850+0000 7f5b27fff700 1 -- 172.21.15.151:0/2093641650 wait complete. 2023-09-30T02:02:56.853 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T02:02:56.865 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2023-09-30T02:02:55.528116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":43136,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":40640,"kb_avail":754931584,"statfs":{"total":773094113280,"available":773049942016,"internally_reserved":0,"allocated":2490368,"data_stored":1519970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41615360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.544219+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:50.544219+0000","last_peered":"2023-09-30T01:57:50.544219+0000","last_clean":"2023-09-30T01:57:50.544219+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:50.544219+0000","last_undegraded":"2023-09-30T01:57:50.544219+0000","last_fullsized":"2023-09-30T01:57:50.544219+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:18:26.843921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111495+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.111495+0000","last_peered":"2023-09-30T01:57:51.111495+0000","last_clean":"2023-09-30T01:57:51.111495+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.111495+0000","last_undegraded":"2023-09-30T01:57:51.111495+0000","last_fullsized":"2023-09-30T01:57:51.111495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111284+0000","last_change":"2023-09-30T01:57:51.106990+0000","last_active":"2023-09-30T01:57:51.111284+0000","last_peered":"2023-09-30T01:57:51.111284+0000","last_clean":"2023-09-30T01:57:51.111284+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T01:57:51.111284+0000","last_undegraded":"2023-09-30T01:57:51.111284+0000","last_fullsized":"2023-09-30T01:57:51.111284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:40.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.551200+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.551200+0000","last_peered":"2023-09-30T01:57:50.551200+0000","last_clean":"2023-09-30T01:57:50.551200+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.551200+0000","last_undegraded":"2023-09-30T01:57:50.551200+0000","last_fullsized":"2023-09-30T01:57:50.551200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.702970+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.702970+0000","last_peered":"2023-09-30T01:57:50.702970+0000","last_clean":"2023-09-30T01:57:50.702970+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.702970+0000","last_undegraded":"2023-09-30T01:57:50.702970+0000","last_fullsized":"2023-09-30T01:57:50.702970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549262+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.549262+0000","last_peered":"2023-09-30T01:57:50.549262+0000","last_clean":"2023-09-30T01:57:50.549262+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.549262+0000","last_undegraded":"2023-09-30T01:57:50.549262+0000","last_fullsized":"2023-09-30T01:57:50.549262+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553788+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.553788+0000","last_peered":"2023-09-30T01:57:50.553788+0000","last_clean":"2023-09-30T01:57:50.553788+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.553788+0000","last_undegraded":"2023-09-30T01:57:50.553788+0000","last_fullsized":"2023-09-30T01:57:50.553788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.129346+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.129346+0000","last_peered":"2023-09-30T01:57:51.129346+0000","last_clean":"2023-09-30T01:57:51.129346+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.129346+0000","last_undegraded":"2023-09-30T01:57:51.129346+0000","last_fullsized":"2023-09-30T01:57:51.129346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.60299999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.0700000000000001,"5min":1.258,"15min":1.258},"last":1.002},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.97699999999999998,"15min":1.01},"min":{"1min":0.64800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2909999999999999,"5min":1.353,"15min":1.353},"last":0.78300000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.64900000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2829999999999999,"5min":1.365,"15min":1.365},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89900000000000002,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.3109999999999999,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.034}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.96399999999999997,"15min":0.92800000000000005},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95299999999999996,"15min":0.97599999999999998},"min":{"1min":0.68000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.2050000000000001,"5min":1.466,"15min":1.466},"last":0.83999999999999997}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.997,"15min":1.0149999999999999},"min":{"1min":0.66600000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.421,"5min":1.446,"15min":1.446},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.0940000000000001,"15min":1.0860000000000001},"min":{"1min":0.878,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.3959999999999999,"5min":1.403,"15min":1.403},"last":1.087}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.92200000000000004,"15min":0.92000000000000004},"min":{"1min":0.67400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0920000000000001,"15min":1.0940000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.456,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.1240000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0429999999999999,"15min":1.052},"min":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.3520000000000001,"5min":1.55,"15min":1.55},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0880000000000001,"15min":1.097},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.93100000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1739999999999999,"15min":1.196},"min":{"1min":0.86299999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.631,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2050000000000001,"15min":1.2250000000000001},"min":{"1min":0.95399999999999996,"5min":0.753,"15min":0.753},"max":{"1min":1.4630000000000001,"5min":1.627,"15min":1.627},"last":0.91400000000000003}]}]},{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91200000000000003,"15min":0.89800000000000002},"min":{"1min":0.58999999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.123},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89100000000000001,"15min":0.91100000000000003},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.141,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.72199999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95599999999999996,"15min":0.91600000000000004},"min":{"1min":0.66700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0409999999999999,"15min":1.0629999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.1479999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91200000000000003},"min":{"1min":0.68799999999999994,"5min":0.63,"15min":0.63},"max":{"1min":1.248,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.66900000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.153,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.999}]},{"osd":3,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95699999999999996,"15min":0.96799999999999997},"min":{"1min":0.68899999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1759999999999999,"5min":1.218,"15min":1.218},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0109999999999999,"15min":1.0209999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.90800000000000003}]},{"osd":4,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90700000000000003,"15min":0.94299999999999995},"min":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2070000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.0669999999999999,"15min":1.101},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.405,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0409999999999999,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4910000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.153},"min":{"1min":0.89600000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.2869999999999999,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.1870000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:53 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.119,"15min":1.115},"min":{"1min":0.89600000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.3640000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.159,"15min":1.1819999999999999},"min":{"1min":0.94699999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.419,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.2569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83799999999999997,"15min":0.83599999999999997},"min":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.98099999999999998,"15min":1.026},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85999999999999999,"15min":0.85599999999999998},"min":{"1min":0.66100000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.262,"15min":1.262},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.92800000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.97899999999999998,"15min":1.028},"min":{"1min":0.66300000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.214,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.876}]},{"osd":3,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.242,"5min":1.575,"15min":1.575},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.95099999999999996},"min":{"1min":0.66900000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.1240000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90700000000000003,"15min":0.89400000000000002},"min":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.028,"5min":1.04,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.216,"5min":1.329,"15min":1.329},"last":1.2010000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1950000000000001,"5min":1.669,"15min":1.669},"last":1.012},{"interface":"front","average":{"1min":1.103,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.2949999999999999,"5min":1.75,"15min":1.75},"last":1.075}]},{"osd":7,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":1.23},"min":{"1min":1.03,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.349,"5min":1.859,"15min":1.859},"last":1.171},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.034,"15min":1.032},"min":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3799999999999999,"5min":1.526,"15min":1.526},"last":1.25}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":0.94099999999999995},"min":{"1min":0.67500000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1970000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1890000000000001,"5min":1.327,"15min":1.327},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.131,"15min":1.167},"min":{"1min":0.96499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.4410000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.1559999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.95699999999999996,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.163,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.73299999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.78600000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.3500000000000001,"5min":1.472,"15min":1.472},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.997,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.4510000000000001,"5min":3.9889999999999999,"15min":3.9889999999999999},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.89900000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.996,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.113},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4099999999999999,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.23}]},{"osd":6,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1020000000000001,"15min":1.048},"min":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.85999999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.2549999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.1819999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1319999999999999,"15min":1.129},"min":{"1min":0.96699999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.5549999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.1519999999999999,"15min":1.1679999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3100000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91200000000000003,"5min":2.29,"15min":2.29},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1659999999999999,"5min":1.569,"15min":1.569},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":1.28,"15min":1.28},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84899999999999998,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.499,"15min":0.499},"max":{"1min":1.3600000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.745}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.0880000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1200000000000001,"5min":1.363,"15min":1.363},"last":1.0249999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0700000000000001,"15min":1.1459999999999999},"min":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.351,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0720000000000001,"15min":1.0840000000000001},"min":{"1min":0.85799999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.532,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.98099999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.046,"15min":1.0800000000000001},"min":{"1min":0.83199999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.474,"5min":1.645,"15min":1.645},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.80800000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.371,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.879}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.2050000000000001,"15min":1.2749999999999999},"min":{"1min":0.79700000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.704,"5min":2.516,"15min":2.516},"last":1.149},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.206,"15min":1.2509999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6679999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":1.002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.524,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3140000000000001},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.4370000000000001,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.1120000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5168,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366672,"statfs":{"total":96636764160,"available":96631472128,"internally_reserved":0,"allocated":172032,"data_stored":56052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65600000000000003,"15min":0.65300000000000002},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.034,"5min":1.181,"15min":1.181},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.73399999999999999,"15min":0.754},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96799999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.86799999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.85899999999999999},"min":{"1min":0.59099999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.401,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.042,"15min":1.0840000000000001},"min":{"1min":0.72499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5580000000000001,"5min":1.819,"15min":1.819},"last":1.091}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0669999999999999,"15min":1.089},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0860000000000001,"15min":1.1060000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.653,"5min":1.659,"15min":1.659},"last":1.002}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.163},"min":{"1min":0.79600000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.667,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.149,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.97999999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1459999999999999,"15min":1.1539999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.224,"15min":1.23},"min":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":1.0629999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.51900000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.0549999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.77500000000000002,"15min":0.76600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97899999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86599999999999999,"5min":1.296,"15min":1.296},"last":0.624},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.87},"min":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1639999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82399999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.874,"15min":0.88500000000000001},"min":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.337,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0449999999999999,"5min":1.218,"15min":1.218},"last":1.04}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97099999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.292,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.093,"5min":1.0920000000000001,"15min":1.119},"min":{"1min":0.79300000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.88800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4910000000000001,"5min":1.625,"15min":1.625},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.135,"15min":1.171},"min":{"1min":0.89400000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.96899999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1539999999999999,"15min":1.2},"min":{"1min":0.86599999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.091,"5min":1.153,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.351,"5min":1.778,"15min":1.778},"last":1.242}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2370000000000001},"min":{"1min":0.95599999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.113},{"interface":"front","average":{"1min":1.151,"5min":1.2170000000000001,"15min":1.262},"min":{"1min":0.96599999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.421,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1819999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":585928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66100000000000003,"15min":0.64600000000000002},"min":{"1min":0.51900000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90900000000000003,"5min":1.006,"15min":1.006},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.76200000000000001,"15min":0.752},"min":{"1min":0.622,"5min":0.441,"15min":0.441},"max":{"1min":1.0349999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.754}]},{"osd":2,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90300000000000002,"5min":1.087,"15min":1.087},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.54500000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71699999999999997,"15min":0.69699999999999995},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.97099999999999997,"5min":1.161,"15min":1.161},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.878},"min":{"1min":0.628,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.056,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0369999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.331,"5min":1.448,"15min":1.448},"last":1.026}]},{"osd":5,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.002,"15min":1.052},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.127,"5min":1.468,"15min":1.468},"last":1.081},{"interface":"front","average":{"1min":1.077,"5min":1.0840000000000001,"15min":1.1160000000000001},"min":{"1min":0.86199999999999999,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.4970000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.165}]},{"osd":6,"last update":"Sat Sep 30 02:02:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0920000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1899999999999999,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.147},"min":{"1min":0.81599999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.5609999999999999,"5min":1.768,"15min":1.768},"last":1.0469999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:01:51 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.155,"15min":1.167},"min":{"1min":0.94799999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.1739999999999999,"15min":1.2090000000000001},"min":{"1min":0.82799999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.296,"5min":1.669,"15min":1.669},"last":1.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:02:56.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:02:56.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 Processor -- start 2023-09-30T02:02:56.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 -- start start 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa9280ab220 0x7fa9280ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928139ed0 con 0x7fa92813aa00 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280ad0e0 con 0x7fa9280ab220 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92e6cf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280a9b00 con 0x7fa9280ab5f0 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52480/0 (socket says 172.21.15.151:52480) 2023-09-30T02:02:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 -- 172.21.15.151:0/436521500 learned_addr learned my addr 172.21.15.151:0/436521500 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92d6cd700 1 --2- 172.21.15.151:0/436521500 >> v2:172.21.15.151:3300/0 conn(0x7fa9280ab220 0x7fa9280ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 -- 172.21.15.151:0/436521500 >> v2:172.21.15.151:3300/0 conn(0x7fa9280ab220 msgr2=0x7fa9280ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 --2- 172.21.15.151:0/436521500 >> v2:172.21.15.151:3300/0 conn(0x7fa9280ab220 0x7fa9280ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 -- 172.21.15.151:0/436521500 >> v1:172.21.15.172:6789/0 conn(0x7fa92813aa00 legacy=0x7fa92813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 -- 172.21.15.151:0/436521500 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280acbf0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa92cecc700 1 --2- 172.21.15.151:0/436521500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa924009f20 tx=0x7fa924009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c549c3efc69c5ec server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa91ffff700 1 -- 172.21.15.151:0/436521500 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92400e070 con 0x7fa9280ab5f0 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa91ffff700 1 -- 172.21.15.151:0/436521500 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92400ba60 con 0x7fa9280ab5f0 2023-09-30T02:02:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.974+0000 7fa91ffff700 1 -- 172.21.15.151:0/436521500 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9240092e0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- 172.21.15.151:0/436521500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 msgr2=0x7fa92813a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/436521500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa924009f20 tx=0x7fa924009840 comp rx=0 tx=0).stop 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- 172.21.15.151:0/436521500 shutdown_connections 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/436521500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab5f0 0x7fa92813a4c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/436521500 >> v2:172.21.15.151:3300/0 conn(0x7fa9280ab220 0x7fa9280ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- 172.21.15.151:0/436521500 >> 172.21.15.151:0/436521500 conn(0x7fa928070510 msgr2=0x7fa928070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- 172.21.15.151:0/436521500 shutdown_connections 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- 172.21.15.151:0/436521500 wait complete. 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 Processor -- start 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- start start 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 0x7fa92812d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280aebd0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928138210 con 0x7fa92813aa00 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92e6cf700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92812da10 con 0x7fa9280ab220 2023-09-30T02:02:56.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92d6cd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53342/0 (socket says 172.21.15.151:53342) 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92d6cd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52486/0 (socket says 172.21.15.151:52486) 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 -- 172.21.15.151:0/3977378734 learned_addr learned my addr 172.21.15.151:0/3977378734 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92dece700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 0x7fa92812d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 -- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 msgr2=0x7fa9280ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 -- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 msgr2=0x7fa92812d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 0x7fa92812d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 -- 172.21.15.151:0/3977378734 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924009ae0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92dece700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 0x7fa92812d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92d6cd700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa92cecc700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa92400b610 tx=0x7fa924018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.978+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9240094e0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa924008930 con 0x7fa9280ab5f0 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9280ad5a0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924008d30 con 0x7fa9280ab5f0 2023-09-30T02:02:56.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9280adac0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa924003040 con 0x7fa9280ab5f0 2023-09-30T02:02:56.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9280008d0 con 0x7fa9280ab5f0 2023-09-30T02:02:56.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.982+0000 7fa91d7fa700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 0x7fa90c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:56.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.986+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fa92407e460 con 0x7fa9280ab5f0 2023-09-30T02:02:56.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.986+0000 7fa92d6cd700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 0x7fa90c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:56.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.986+0000 7fa92d6cd700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 0x7fa90c062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa918011480 tx=0x7fa9180091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:56.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:56.990+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa92404ecf0 con 0x7fa9280ab5f0 2023-09-30T02:02:57.158 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9280aae20 con 0x7fa9280ab5f0 2023-09-30T02:02:57.159 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa91d7fa700 1 -- 172.21.15.151:0/3977378734 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13695 (secure 0 0 0) 0x7fa924052310 con 0x7fa9280ab5f0 2023-09-30T02:02:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:02:57.159 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":18,"fsid":"70a77e99-16f3-4896-b121-bcea9198ad08","created":"2023-09-30T01:57:37.276087+0000","modified":"2023-09-30T01:57:51.540012+0000","last_up_change":"2023-09-30T01:57:47.500496+0000","last_in_change":"2023-09-30T01:57:40.080437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:57:47.499184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:57:48.763412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8e644ce0-bab6-470f-8040-4b28674cfa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":31361},{"type":"v1","addr":"172.21.15.151:6818","nonce":31361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":31361},{"type":"v1","addr":"172.21.15.151:6824","nonce":31361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":31361},{"type":"v1","addr":"172.21.15.151:6831","nonce":31361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":31361},{"type":"v1","addr":"172.21.15.151:6828","nonce":31361}]},"public_addr":"172.21.15.151:6818/31361","cluster_addr":"172.21.15.151:6824/31361","heartbeat_back_addr":"172.21.15.151:6831/31361","heartbeat_front_addr":"172.21.15.151:6828/31361","state":["exists","up"]},{"osd":1,"uuid":"cff33849-65d5-4e21-a3f2-ac9f2aed071a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":31363},{"type":"v1","addr":"172.21.15.151:6811","nonce":31363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":31363},{"type":"v1","addr":"172.21.15.151:6815","nonce":31363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":31363},{"type":"v1","addr":"172.21.15.151:6823","nonce":31363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":31363},{"type":"v1","addr":"172.21.15.151:6819","nonce":31363}]},"public_addr":"172.21.15.151:6811/31363","cluster_addr":"172.21.15.151:6815/31363","heartbeat_back_addr":"172.21.15.151:6823/31363","heartbeat_front_addr":"172.21.15.151:6819/31363","state":["exists","up"]},{"osd":2,"uuid":"8a7cdb0c-0d40-4d29-bff5-0a76bf409f0f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":31362},{"type":"v1","addr":"172.21.15.151:6810","nonce":31362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":31362},{"type":"v1","addr":"172.21.15.151:6816","nonce":31362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6827","nonce":31362},{"type":"v1","addr":"172.21.15.151:6829","nonce":31362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":31362},{"type":"v1","addr":"172.21.15.151:6825","nonce":31362}]},"public_addr":"172.21.15.151:6810/31362","cluster_addr":"172.21.15.151:6816/31362","heartbeat_back_addr":"172.21.15.151:6829/31362","heartbeat_front_addr":"172.21.15.151:6825/31362","state":["exists","up"]},{"osd":3,"uuid":"fd981572-4e9d-4d6c-8c40-5e7bd24eadc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":31360},{"type":"v1","addr":"172.21.15.151:6801","nonce":31360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":31360},{"type":"v1","addr":"172.21.15.151:6803","nonce":31360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":31360},{"type":"v1","addr":"172.21.15.151:6807","nonce":31360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":31360},{"type":"v1","addr":"172.21.15.151:6805","nonce":31360}]},"public_addr":"172.21.15.151:6801/31360","cluster_addr":"172.21.15.151:6803/31360","heartbeat_back_addr":"172.21.15.151:6807/31360","heartbeat_front_addr":"172.21.15.151:6805/31360","state":["exists","up"]},{"osd":4,"uuid":"0ac4613b-f5e3-475e-a5d5-71175f061a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":31451},{"type":"v1","addr":"172.21.15.172:6801","nonce":31451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":31451},{"type":"v1","addr":"172.21.15.172:6803","nonce":31451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":31451},{"type":"v1","addr":"172.21.15.172:6807","nonce":31451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":31451},{"type":"v1","addr":"172.21.15.172:6805","nonce":31451}]},"public_addr":"172.21.15.172:6801/31451","cluster_addr":"172.21.15.172:6803/31451","heartbeat_back_addr":"172.21.15.172:6807/31451","heartbeat_front_addr":"172.21.15.172:6805/31451","state":["exists","up"]},{"osd":5,"uuid":"d833c90c-9efc-4367-8b92-b01219b686e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":31452},{"type":"v1","addr":"172.21.15.172:6811","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":31452},{"type":"v1","addr":"172.21.15.172:6814","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":31452},{"type":"v1","addr":"172.21.15.172:6820","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":31452},{"type":"v1","addr":"172.21.15.172:6817","nonce":31452}]},"public_addr":"172.21.15.172:6811/31452","cluster_addr":"172.21.15.172:6814/31452","heartbeat_back_addr":"172.21.15.172:6820/31452","heartbeat_front_addr":"172.21.15.172:6817/31452","state":["exists","up"]},{"osd":6,"uuid":"b54db1f9-34f1-4ef9-96a1-fde5a9555808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31453},{"type":"v1","addr":"172.21.15.172:6825","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":31453},{"type":"v1","addr":"172.21.15.172:6827","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":31453},{"type":"v1","addr":"172.21.15.172:6831","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":31453},{"type":"v1","addr":"172.21.15.172:6829","nonce":31453}]},"public_addr":"172.21.15.172:6825/31453","cluster_addr":"172.21.15.172:6827/31453","heartbeat_back_addr":"172.21.15.172:6831/31453","heartbeat_front_addr":"172.21.15.172:6829/31453","state":["exists","up"]},{"osd":7,"uuid":"24126a91-6681-47fb-87f8-490864058d02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":31450},{"type":"v1","addr":"172.21.15.172:6809","nonce":31450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":31450},{"type":"v1","addr":"172.21.15.172:6815","nonce":31450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":31450},{"type":"v1","addr":"172.21.15.172:6823","nonce":31450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":31450},{"type":"v1","addr":"172.21.15.172:6821","nonce":31450}]},"public_addr":"172.21.15.172:6809/31450","cluster_addr":"172.21.15.172:6815/31450","heartbeat_back_addr":"172.21.15.172:6823/31450","heartbeat_front_addr":"172.21.15.172:6821/31450","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.403867+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.433944+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.158817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.278191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.087404+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.063855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.494600+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:57:45.286608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:02:57.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 msgr2=0x7fa90c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 0x7fa90c062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa918011480 tx=0x7fa9180091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 msgr2=0x7fa928137cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa92400b610 tx=0x7fa924018000 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 shutdown_connections 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fa90c060550 0x7fa90c062a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa92813aa00 0x7fa92812d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa9280ab5f0 0x7fa928137cd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 --2- 172.21.15.151:0/3977378734 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa9280ab220 0x7fa9280ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:57.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 >> 172.21.15.151:0/3977378734 conn(0x7fa928070510 msgr2=0x7fa928131400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:57.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 shutdown_connections 2023-09-30T02:02:57.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:57.158+0000 7fa92e6cf700 1 -- 172.21.15.151:0/3977378734 wait complete. 2023-09-30T02:02:58.173 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T02:02:58.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:02:58.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.294+0000 7fbe26c2a700 1 Processor -- start 2023-09-30T02:02:58.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.294+0000 7fbe26c2a700 1 -- start start 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbe20140b70 0x7fbe20140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe200aa330 con 0x7fbe20070740 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe200a9510 con 0x7fbe20140b70 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2006f5f0 con 0x7fbe2013d650 2023-09-30T02:02:58.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26429700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26429700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52492/0 (socket says 172.21.15.151:52492) 2023-09-30T02:02:58.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26429700 1 -- 172.21.15.151:0/2779633016 learned_addr learned my addr 172.21.15.151:0/2779633016 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 --2- 172.21.15.151:0/2779633016 >> v2:172.21.15.151:3300/0 conn(0x7fbe20140b70 0x7fbe20140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe24c26700 1 -- 172.21.15.151:0/2779633016 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086206276 0 0) 0x7fbe200aa330 con 0x7fbe20070740 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe24c26700 1 -- 172.21.15.151:0/2779633016 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c003570 con 0x7fbe20070740 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 -- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 msgr2=0x7fbe2013dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 -- 172.21.15.151:0/2779633016 >> v1:172.21.15.151:6789/0 conn(0x7fbe20070740 legacy=0x7fbe200a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 -- 172.21.15.151:0/2779633016 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20070b10 con 0x7fbe20140b70 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26429700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe25427700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20140fd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0098e0 tx=0x7fbe1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fb75cdb1fa8be7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe24c26700 1 -- 172.21.15.151:0/2779633016 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c011630 con 0x7fbe20140b70 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe24c26700 1 -- 172.21.15.151:0/2779633016 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c011790 con 0x7fbe20140b70 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe24c26700 1 -- 172.21.15.151:0/2779633016 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c011b30 con 0x7fbe20140b70 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 msgr2=0x7fbe20140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20140fd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0098e0 tx=0x7fbe1c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- 172.21.15.151:0/2779633016 shutdown_connections 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe2013dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/2779633016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20140fd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- 172.21.15.151:0/2779633016 >> 172.21.15.151:0/2779633016 conn(0x7fbe20067710 msgr2=0x7fbe20067b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- 172.21.15.151:0/2779633016 shutdown_connections 2023-09-30T02:02:58.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.298+0000 7fbe26c2a700 1 -- 172.21.15.151:0/2779633016 wait complete. 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 Processor -- start 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 -- start start 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe20070740 0x7fbe20117e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe201503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20118810 con 0x7fbe20140b70 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2006da30 con 0x7fbe20070740 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26c2a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe201183c0 con 0x7fbe2013d650 2023-09-30T02:02:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe25427700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe201503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53348/0 (socket says 172.21.15.151:53348) 2023-09-30T02:02:58.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 -- 172.21.15.151:0/1178854279 learned_addr learned my addr 172.21.15.151:0/1178854279 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe25c28700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe20070740 0x7fbe20117e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 -- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 msgr2=0x7fbe201503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe201503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 -- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe20070740 msgr2=0x7fbe20117e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbe20070740 0x7fbe20117e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18009ae0 con 0x7fbe20140b70 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe25427700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe2013d650 0x7fbe201503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:58.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.302+0000 7fbe26429700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20150900 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fd0 tx=0x7fbe1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.306+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18018070 con 0x7fbe20140b70 2023-09-30T02:02:58.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.306+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1800f040 con 0x7fbe20140b70 2023-09-30T02:02:58.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.306+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c009420 con 0x7fbe20140b70 2023-09-30T02:02:58.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.306+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1800cbf0 con 0x7fbe20140b70 2023-09-30T02:02:58.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.306+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20150fe0 con 0x7fbe20140b70 2023-09-30T02:02:58.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe200670a0 con 0x7fbe20140b70 2023-09-30T02:02:58.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fbe18003020 con 0x7fbe20140b70 2023-09-30T02:02:58.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbe04060580 0x7fbe04062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fbe1807c420 con 0x7fbe20140b70 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe25c28700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbe04060580 0x7fbe04062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7fbe04066950 0x7fbe04068d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe04069410 con 0x7fbe04066950 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe1807ea60 con 0x7fbe20140b70 2023-09-30T02:02:58.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe25427700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7fbe04066950 0x7fbe04068d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe25427700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7fbe04066950 0x7fbe04068d60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe25c28700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbe04060580 0x7fbe04062a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe10007fd0 tx=0x7fbe10007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.310+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== osd.0 v2:172.21.15.151:6814/31361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbe04069410 con 0x7fbe04066950 2023-09-30T02:02:58.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 --> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbe200a91c0 con 0x7fbe04066950 2023-09-30T02:02:58.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe15ffb700 1 -- 172.21.15.151:0/1178854279 <== osd.0 v2:172.21.15.151:6814/31361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fbe200a91c0 con 0x7fbe04066950 2023-09-30T02:02:58.329 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:02:58.329 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:02:58.329 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:02:58.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7fbe04066950 msgr2=0x7fbe04068d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:6814/31361,v1:172.21.15.151:6818/31361] conn(0x7fbe04066950 0x7fbe04068d60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbe04060580 msgr2=0x7fbe04062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fbe04060580 0x7fbe04062a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe10007fd0 tx=0x7fbe10007d50 comp rx=0 tx=0).stop 2023-09-30T02:02:58.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 msgr2=0x7fbe20150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 --2- 172.21.15.151:0/1178854279 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe20140b70 0x7fbe20150900 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009fd0 tx=0x7fbe1800b040 comp rx=0 tx=0).stop 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26429700 1 -- 172.21.15.151:0/1178854279 reap_dead start 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 shutdown_connections 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 >> 172.21.15.151:0/1178854279 conn(0x7fbe20067710 msgr2=0x7fbe2013dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 shutdown_connections 2023-09-30T02:02:58.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.326+0000 7fbe26c2a700 1 -- 172.21.15.151:0/1178854279 wait complete. 2023-09-30T02:02:58.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T02:02:58.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 Processor -- start 2023-09-30T02:02:58.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- start start 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2aa0143bd0 0x7f2aa0145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00ab610 con 0x7f2aa0070740 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa0070b10 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00ac160 con 0x7f2aa0143bd0 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9effd700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f2aa0070740 0x7f2aa013e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48058/0 (socket says 172.21.15.151:48058) 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9effd700 1 -- 172.21.15.151:0/1962423043 learned_addr learned my addr 172.21.15.151:0/1962423043 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:58.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9f7fe700 1 --2- 172.21.15.151:0/1962423043 >> v2:172.21.15.151:3300/0 conn(0x7f2aa0143bd0 0x7f2aa0145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9dffb700 1 -- 172.21.15.151:0/1962423043 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742142894 0 0) 0x7f2aa00ab610 con 0x7f2aa0070740 2023-09-30T02:02:58.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9dffb700 1 -- 172.21.15.151:0/1962423043 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88003570 con 0x7f2aa0070740 2023-09-30T02:02:58.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9f7fe700 1 -- 172.21.15.151:0/1962423043 >> v1:172.21.15.172:6789/0 conn(0x7f2aa00ab0f0 legacy=0x7f2aa013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9f7fe700 1 -- 172.21.15.151:0/1962423043 >> v1:172.21.15.151:6789/0 conn(0x7f2aa0070740 legacy=0x7f2aa013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9f7fe700 1 -- 172.21.15.151:0/1962423043 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0068160 con 0x7f2aa0143bd0 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9f7fe700 1 --2- 172.21.15.151:0/1962423043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0143bd0 0x7f2aa0145fb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009e60 tx=0x7f2a98009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e69b516a270802 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9dffb700 1 -- 172.21.15.151:0/1962423043 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800e070 con 0x7f2aa0143bd0 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9dffb700 1 -- 172.21.15.151:0/1962423043 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9800bb60 con 0x7f2aa0143bd0 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9dffb700 1 -- 172.21.15.151:0/1962423043 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98012430 con 0x7f2aa0143bd0 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- 172.21.15.151:0/1962423043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0143bd0 msgr2=0x7f2aa0145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/1962423043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0143bd0 0x7f2aa0145fb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009e60 tx=0x7f2a98009780 comp rx=0 tx=0).stop 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- 172.21.15.151:0/1962423043 shutdown_connections 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/1962423043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0143bd0 0x7f2aa0145fb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- 172.21.15.151:0/1962423043 >> 172.21.15.151:0/1962423043 conn(0x7f2aa00b0400 msgr2=0x7f2aa00b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- 172.21.15.151:0/1962423043 shutdown_connections 2023-09-30T02:02:58.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.450+0000 7f2a9ffff700 1 -- 172.21.15.151:0/1962423043 wait complete. 2023-09-30T02:02:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 Processor -- start 2023-09-30T02:02:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- start start 2023-09-30T02:02:58.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 0x7f2aa0131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa0143bd0 0x7f2aa014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0131da0 con 0x7f2aa0070740 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa014a960 con 0x7f2aa0143bd0 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0138fc0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9effd700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 0x7f2aa0131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52504/0 (socket says 172.21.15.151:52504) 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 -- 172.21.15.151:0/3272167885 learned_addr learned my addr 172.21.15.151:0/3272167885 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9f7fe700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa0143bd0 0x7f2aa014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 -- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa0143bd0 msgr2=0x7f2aa014b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa0143bd0 0x7f2aa014b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 -- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 msgr2=0x7f2aa0131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 0x7f2aa0131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 -- 172.21.15.151:0/3272167885 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98009a20 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9effd700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 0x7f2aa0131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9e7fc700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c790 tx=0x7f2a94007f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94011590 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94011710 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94011ad0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01392a0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.454+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa01397c0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa00008d0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2a9401b3b0 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a86ffd700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 0x7f2a8c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2a9407b970 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a9effd700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 0x7f2a8c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.458+0000 7f2a9effd700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 0x7f2a8c062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a90007fd0 tx=0x7f2a90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.462+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2a9404c150 con 0x7f2aa00ab0f0 2023-09-30T02:02:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.622+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aa0139100 con 0x7f2a8c060550 2023-09-30T02:02:58.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.622+0000 7f2a86ffd700 1 -- 172.21.15.151:0/3272167885 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2aa0139100 con 0x7f2a8c060550 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 msgr2=0x7f2a8c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 0x7f2a8c062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a90007fd0 tx=0x7f2a90009040 comp rx=0 tx=0).stop 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 msgr2=0x7f2aa014ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c790 tx=0x7f2a94007f40 comp rx=0 tx=0).stop 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 shutdown_connections 2023-09-30T02:02:58.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2a8c060550 0x7f2a8c062a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2aa0143bd0 0x7f2aa014b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2aa00ab0f0 0x7f2aa014ae80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 --2- 172.21.15.151:0/3272167885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2aa0070740 0x7f2aa0131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 >> 172.21.15.151:0/3272167885 conn(0x7f2aa00b0400 msgr2=0x7f2aa00ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 shutdown_connections 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.626+0000 7f2a9ffff700 1 -- 172.21.15.151:0/3272167885 wait complete. 2023-09-30T02:02:58.629 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T02:02:58.639 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T02:02:58.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:02:58.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.750+0000 7fea72ba8700 1 Processor -- start 2023-09-30T02:02:58.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.750+0000 7fea72ba8700 1 -- start start 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea72ba8700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fea6c070740 0x7fea6c0a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea72ba8700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c140820 con 0x7fea6c06ea90 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea72ba8700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c070b10 con 0x7fea6c140cb0 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea72ba8700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c13e200 con 0x7fea6c070740 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea723a7700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fea6c06ea90 0x7fea6c06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48062/0 (socket says 172.21.15.151:48062) 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea723a7700 1 -- 172.21.15.151:0/2122839922 learned_addr learned my addr 172.21.15.151:0/2122839922 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 --2- 172.21.15.151:0/2122839922 >> v2:172.21.15.172:3300/0 conn(0x7fea6c070740 0x7fea6c0a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44218737 0 0) 0x7fea6c140820 con 0x7fea6c06ea90 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58003570 con 0x7fea6c06ea90 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943795577 0 0) 0x7fea58003570 con 0x7fea6c06ea90 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 >> v2:172.21.15.172:3300/0 conn(0x7fea6c070740 msgr2=0x7fea6c0a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 --2- 172.21.15.151:0/2122839922 >> v2:172.21.15.172:3300/0 conn(0x7fea6c070740 0x7fea6c0a8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 >> v1:172.21.15.172:6789/0 conn(0x7fea6c140cb0 legacy=0x7fea6c1410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0ac5c0 con 0x7fea6c06ea90 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7fea640017f0 con 0x7fea6c06ea90 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 >> v1:172.21.15.151:6789/0 conn(0x7fea6c06ea90 legacy=0x7fea6c06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 0x7fea58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58004810 0x7fea58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea580051c0 0x7fea5800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c13e200 con 0x7fea58003a60 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800c160 con 0x7fea58004810 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800c320 con 0x7fea580051c0 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 0x7fea58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea723a7700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea580051c0 0x7fea5800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea713a5700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58004810 0x7fea58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 -- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea580051c0 msgr2=0x7fea5800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea580051c0 0x7fea5800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 -- 172.21.15.151:0/2122839922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58004810 msgr2=0x7fea58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58004810 0x7fea58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea723a7700 1 -- 172.21.15.151:0/2122839922 reap_dead start 2023-09-30T02:02:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 -- 172.21.15.151:0/2122839922 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58003570 con 0x7fea58003a60 2023-09-30T02:02:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea71ba6700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 0x7fea58003e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fea68009c00 tx=0x7fea680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2652258f1cf7b2f1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800e710 con 0x7fea58003a60 2023-09-30T02:02:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800e870 con 0x7fea58003a60 2023-09-30T02:02:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.754+0000 7fea70ba4700 1 -- 172.21.15.151:0/2122839922 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800ec30 con 0x7fea58003a60 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 msgr2=0x7fea58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 0x7fea58003e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fea68009c00 tx=0x7fea680091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- 172.21.15.151:0/2122839922 shutdown_connections 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 --2- 172.21.15.151:0/2122839922 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea58003a60 0x7fea58003e70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- 172.21.15.151:0/2122839922 >> 172.21.15.151:0/2122839922 conn(0x7fea6c067710 msgr2=0x7fea6c067b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- 172.21.15.151:0/2122839922 shutdown_connections 2023-09-30T02:02:58.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- 172.21.15.151:0/2122839922 wait complete. 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 Processor -- start 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- start start 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58003a60 0x7fea6c06df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 0x7fea6c14cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea6c14d330 0x7fea6c14fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c14c960 con 0x7fea6c14d330 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c14d7b0 con 0x7fea58003a60 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.758+0000 7fea72ba8700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1195b0 con 0x7fea6c1181f0 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea713a5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 0x7fea6c14cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea713a5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 0x7fea6c14cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52520/0 (socket says 172.21.15.151:52520) 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea713a5700 1 -- 172.21.15.151:0/1498207592 learned_addr learned my addr 172.21.15.151:0/1498207592 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58003a60 0x7fea6c06df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea723a7700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea6c14d330 0x7fea6c14fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 -- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 msgr2=0x7fea6c14cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 0x7fea6c14cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 -- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea6c14d330 msgr2=0x7fea6c14fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea6c14d330 0x7fea6c14fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c06e800 con 0x7fea58003a60 2023-09-30T02:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea713a5700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea6c1181f0 0x7fea6c14cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:58.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea71ba6700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58003a60 0x7fea6c06df20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea68019000 tx=0x7fea6801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68022070 con 0x7fea58003a60 2023-09-30T02:02:58.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800e9e0 con 0x7fea58003a60 2023-09-30T02:02:58.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800e710 con 0x7fea58003a60 2023-09-30T02:02:58.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c06d0b0 con 0x7fea58003a60 2023-09-30T02:02:58.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c1503a0 con 0x7fea58003a60 2023-09-30T02:02:58.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.762+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fea6c06b9d0 con 0x7fea58003a60 2023-09-30T02:02:58.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fea6801c660 con 0x7fea58003a60 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fea50060550 0x7fea50062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fea6807e5c0 con 0x7fea58003a60 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7fea50066920 0x7fea50068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea500693e0 con 0x7fea50066920 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea6807e990 con 0x7fea58003a60 2023-09-30T02:02:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea713a5700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fea50060550 0x7fea50062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea723a7700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7fea50066920 0x7fea50068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea723a7700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7fea50066920 0x7fea50068d30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea713a5700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fea50060550 0x7fea50062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea6c06cff0 tx=0x7fea60007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.766+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== osd.1 v2:172.21.15.151:6809/31363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fea500693e0 con 0x7fea50066920 2023-09-30T02:02:58.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 --> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fea6c119720 con 0x7fea50066920 2023-09-30T02:02:58.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea5dffb700 1 -- 172.21.15.151:0/1498207592 <== osd.1 v2:172.21.15.151:6809/31363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fea6c119720 con 0x7fea50066920 2023-09-30T02:02:58.785 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:02:58.785 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:02:58.785 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7fea50066920 msgr2=0x7fea50068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.151:6809/31363,v1:172.21.15.151:6811/31363] conn(0x7fea50066920 0x7fea50068d30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fea50060550 msgr2=0x7fea50062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fea50060550 0x7fea50062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea6c06cff0 tx=0x7fea60007d30 comp rx=0 tx=0).stop 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58003a60 msgr2=0x7fea6c06df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 --2- 172.21.15.151:0/1498207592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea58003a60 0x7fea6c06df20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea68019000 tx=0x7fea6801a000 comp rx=0 tx=0).stop 2023-09-30T02:02:58.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea723a7700 1 -- 172.21.15.151:0/1498207592 reap_dead start 2023-09-30T02:02:58.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 shutdown_connections 2023-09-30T02:02:58.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 >> 172.21.15.151:0/1498207592 conn(0x7fea6c067710 msgr2=0x7fea6c0ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 shutdown_connections 2023-09-30T02:02:58.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.782+0000 7fea72ba8700 1 -- 172.21.15.151:0/1498207592 wait complete. 2023-09-30T02:02:58.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T02:02:58.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 Processor -- start 2023-09-30T02:02:58.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- start start 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3aa80accd0 0x7f3aa813cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa813d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 0x7f3aa80aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa80ad0a0 con 0x7f3aa80accd0 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa80abfc0 con 0x7f3aa80aa110 2023-09-30T02:02:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa806ce70 con 0x7f3aa806ea10 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3aa80accd0 0x7f3aa813cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aa7fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa813d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3aa80accd0 0x7f3aa813cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53388/0 (socket says 172.21.15.151:53388) 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 -- 172.21.15.151:0/304784369 learned_addr learned my addr 172.21.15.151:0/304784369 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aad53c700 1 --2- 172.21.15.151:0/304784369 >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 0x7f3aa80aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 -- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 msgr2=0x7f3aa813d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa813d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 -- 172.21.15.151:0/304784369 >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 msgr2=0x7f3aa80aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/304784369 >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 0x7f3aa80aa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 -- 172.21.15.151:0/304784369 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa806bc10 con 0x7f3aa80accd0 2023-09-30T02:02:58.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aad53c700 1 --2- 172.21.15.151:0/304784369 >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 0x7f3aa80aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:58.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80accd0 0x7f3aa813cc90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80abd40 tx=0x7f3a90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59973ee9caeb5023 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aa77fe700 1 -- 172.21.15.151:0/304784369 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90009e70 con 0x7f3aa80accd0 2023-09-30T02:02:58.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aa77fe700 1 -- 172.21.15.151:0/304784369 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9000b8f0 con 0x7f3aa80accd0 2023-09-30T02:02:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80accd0 msgr2=0x7f3aa813cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80accd0 0x7f3aa813cc90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80abd40 tx=0x7f3a90009250 comp rx=0 tx=0).stop 2023-09-30T02:02:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- 172.21.15.151:0/304784369 shutdown_connections 2023-09-30T02:02:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/304784369 >> v2:172.21.15.172:3300/0 conn(0x7f3aa80aa110 0x7f3aa80aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa813d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/304784369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80accd0 0x7f3aa813cc90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- 172.21.15.151:0/304784369 >> 172.21.15.151:0/304784369 conn(0x7f3aa8070510 msgr2=0x7f3aa8070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.906+0000 7f3aadd3d700 1 -- 172.21.15.151:0/304784369 shutdown_connections 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- 172.21.15.151:0/304784369 wait complete. 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 Processor -- start 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- start start 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 0x7f3aa8134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80accd0 0x7f3aa8135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80a6a60 con 0x7f3aa80aa110 2023-09-30T02:02:58.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8135640 con 0x7f3aa80accd0 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80af5b0 con 0x7f3aa806ea10 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aa7fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 0x7f3aa8134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52536/0 (socket says 172.21.15.151:52536) 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 -- 172.21.15.151:0/3302415383 learned_addr learned my addr 172.21.15.151:0/3302415383 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aad53c700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80accd0 0x7f3aa8135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 -- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80accd0 msgr2=0x7f3aa8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80accd0 0x7f3aa8135100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 -- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 msgr2=0x7f3aa8134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 0x7f3aa8134bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 -- 172.21.15.151:0/3302415383 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900094f0 con 0x7f3aa806ea10 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aa7fff700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 0x7f3aa8134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aacd3b700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a900094c0 tx=0x7f3a90017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9001a950 con 0x7f3aa806ea10 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9001aab0 con 0x7f3aa806ea10 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa80a76c0 con 0x7f3aa806ea10 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9001ae70 con 0x7f3aa806ea10 2023-09-30T02:02:58.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.910+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa80a6590 con 0x7f3aa806ea10 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa80a6ba0 con 0x7f3aa806ea10 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3a90023070 con 0x7f3aa806ea10 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aa4ff9700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 0x7f3a88062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3a9007d650 con 0x7f3aa806ea10 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aa7fff700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 0x7f3a88062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:58.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.914+0000 7f3aa7fff700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 0x7f3a88062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80afea0 tx=0x7f3aa000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:58.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:58.922+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3a9004dd90 con 0x7f3aa806ea10 2023-09-30T02:02:59.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.086+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3aa80af740 con 0x7f3a88060550 2023-09-30T02:02:59.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.086+0000 7f3aa4ff9700 1 -- 172.21.15.151:0/3302415383 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3aa80af740 con 0x7f3a88060550 2023-09-30T02:02:59.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 msgr2=0x7f3a88062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 0x7f3a88062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80afea0 tx=0x7f3aa000b040 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 msgr2=0x7f3aa80a7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a900094c0 tx=0x7f3a90017000 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 shutdown_connections 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3a88060550 0x7f3a88062a00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3aa80accd0 0x7f3aa8135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3aa80aa110 0x7f3aa8134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 --2- 172.21.15.151:0/3302415383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3aa806ea10 0x7f3aa80a7180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 >> 172.21.15.151:0/3302415383 conn(0x7f3aa8070510 msgr2=0x7f3aa8136e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 shutdown_connections 2023-09-30T02:02:59.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.090+0000 7f3aadd3d700 1 -- 172.21.15.151:0/3302415383 wait complete. 2023-09-30T02:02:59.095 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T02:02:59.109 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T02:02:59.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 Processor -- start 2023-09-30T02:02:59.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 -- start start 2023-09-30T02:02:59.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8cbc0b0ec0 0x7f8cbc0a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0af940 con 0x7f8cbc0abce0 2023-09-30T02:02:59.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cbc0aa820 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc2a17700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc0aad80 con 0x7f8cbc0ab360 2023-09-30T02:02:59.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1214700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1a15700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8cbc0b0ec0 0x7f8cbc0a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1214700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52548/0 (socket says 172.21.15.151:52548) 2023-09-30T02:02:59.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1214700 1 -- 172.21.15.151:0/2035687363 learned_addr learned my addr 172.21.15.151:0/2035687363 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1a15700 1 -- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 msgr2=0x7f8cbc0ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1a15700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1a15700 1 -- 172.21.15.151:0/2035687363 >> v1:172.21.15.151:6789/0 conn(0x7f8cbc0abce0 legacy=0x7f8cbc0ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1a15700 1 -- 172.21.15.151:0/2035687363 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0a9870 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.222+0000 7f8cc1214700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc1a15700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc0a91a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0098e0 tx=0x7f8cac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4841420447a671de server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc0a13700 1 -- 172.21.15.151:0/2035687363 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00b640 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc0a13700 1 -- 172.21.15.151:0/2035687363 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00b7a0 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc0a13700 1 -- 172.21.15.151:0/2035687363 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00bb40 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 msgr2=0x7f8cbc0a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc0a91a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0098e0 tx=0x7f8cac0091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- 172.21.15.151:0/2035687363 shutdown_connections 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc0ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/2035687363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc0a91a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- 172.21.15.151:0/2035687363 >> 172.21.15.151:0/2035687363 conn(0x7f8cbc0b4670 msgr2=0x7f8cbc0b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- 172.21.15.151:0/2035687363 shutdown_connections 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- 172.21.15.151:0/2035687363 wait complete. 2023-09-30T02:02:59.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 Processor -- start 2023-09-30T02:02:59.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.226+0000 7f8cc2a17700 1 -- start start 2023-09-30T02:02:59.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0abce0 0x7f8cbc138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc06ae30 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc13d6a0 con 0x7f8cbc0abce0 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1397a0 con 0x7f8cbc0ab360 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc1a15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc1a15700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52554/0 (socket says 172.21.15.151:52554) 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc139260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53424/0 (socket says 172.21.15.151:53424) 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc1a15700 1 -- 172.21.15.151:0/637291371 learned_addr learned my addr 172.21.15.151:0/637291371 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc1214700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0abce0 0x7f8cbc138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 -- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 msgr2=0x7f8cbc150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8cbc0ab360 0x7f8cbc150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 -- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0abce0 msgr2=0x7f8cbc138d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cbc0abce0 0x7f8cbc138d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80094f0 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2216700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc139260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c830 tx=0x7f8cb4001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4012630 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb4012790 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac009420 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4012b50 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc151300 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.230+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cbc0aa460 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8cb4011380 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ca0060550 0x7f8ca0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8cb407c6f0 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cc1a15700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ca0060550 0x7f8ca0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7f8ca0066920 0x7f8ca0068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ca00693e0 con 0x7f8ca0066920 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb407ed00 con 0x7f8cbc0b0ec0 2023-09-30T02:02:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cc1214700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7f8ca0066920 0x7f8ca0068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cc1a15700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ca0060550 0x7f8ca0062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c010 tx=0x7f8cac017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cc1214700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7f8ca0066920 0x7f8ca0068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.234+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== osd.2 v2:172.21.15.151:6808/31362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8ca00693e0 con 0x7f8ca0066920 2023-09-30T02:02:59.252 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 --> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8cbc0670a0 con 0x7f8ca0066920 2023-09-30T02:02:59.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cb1ffb700 1 -- 172.21.15.151:0/637291371 <== osd.2 v2:172.21.15.151:6808/31362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8cbc0670a0 con 0x7f8ca0066920 2023-09-30T02:02:59.253 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:02:59.253 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:02:59.253 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:02:59.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7f8ca0066920 msgr2=0x7f8ca0068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:6808/31362,v1:172.21.15.151:6810/31362] conn(0x7f8ca0066920 0x7f8ca0068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ca0060550 msgr2=0x7f8ca0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8ca0060550 0x7f8ca0062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c010 tx=0x7f8cac017000 comp rx=0 tx=0).stop 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 msgr2=0x7f8cbc139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 --2- 172.21.15.151:0/637291371 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8cbc0b0ec0 0x7f8cbc139260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c830 tx=0x7f8cb4001690 comp rx=0 tx=0).stop 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2216700 1 -- 172.21.15.151:0/637291371 reap_dead start 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 shutdown_connections 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 >> 172.21.15.151:0/637291371 conn(0x7f8cbc0b4670 msgr2=0x7f8cbc0b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 shutdown_connections 2023-09-30T02:02:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.250+0000 7f8cc2a17700 1 -- 172.21.15.151:0/637291371 wait complete. 2023-09-30T02:02:59.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T02:02:59.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 Processor -- start 2023-09-30T02:02:59.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 -- start start 2023-09-30T02:02:59.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f533406b460 0x7f533406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340b02f0 con 0x7f533406b830 2023-09-30T02:02:59.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340b0e90 con 0x7f5334149520 2023-09-30T02:02:59.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5338b2a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533406cd90 con 0x7f533406b460 2023-09-30T02:02:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f533259c700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f533406b460 0x7f533406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5329d9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f533406b830 0x7f53340b0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48084/0 (socket says 172.21.15.151:48084) 2023-09-30T02:02:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.370+0000 7f5329d9b700 1 -- 172.21.15.151:0/2980858242 learned_addr learned my addr 172.21.15.151:0/2980858242 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5331d9b700 1 -- 172.21.15.151:0/2980858242 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201525616 0 0) 0x7f53340b02f0 con 0x7f533406b830 2023-09-30T02:02:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f533259c700 1 -- 172.21.15.151:0/2980858242 >> v1:172.21.15.172:6789/0 conn(0x7f5334149520 legacy=0x7f533414b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f533259c700 1 -- 172.21.15.151:0/2980858242 >> v1:172.21.15.151:6789/0 conn(0x7f533406b830 legacy=0x7f53340b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f533259c700 1 -- 172.21.15.151:0/2980858242 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533406d070 con 0x7f533406b460 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f533259c700 1 --2- 172.21.15.151:0/2980858242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b460 0x7f533406d490 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53240098e0 tx=0x7f53240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee96b08a13e96cc2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5331d9b700 1 -- 172.21.15.151:0/2980858242 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324009e80 con 0x7f533406b460 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5331d9b700 1 -- 172.21.15.151:0/2980858242 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532400b860 con 0x7f533406b460 2023-09-30T02:02:59.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5331d9b700 1 -- 172.21.15.151:0/2980858242 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532400bbe0 con 0x7f533406b460 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- 172.21.15.151:0/2980858242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b460 msgr2=0x7f533406d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2980858242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b460 0x7f533406d490 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53240098e0 tx=0x7f53240091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- 172.21.15.151:0/2980858242 shutdown_connections 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2980858242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b460 0x7f533406d490 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- 172.21.15.151:0/2980858242 >> 172.21.15.151:0/2980858242 conn(0x7f5334070510 msgr2=0x7f5334070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- 172.21.15.151:0/2980858242 shutdown_connections 2023-09-30T02:02:59.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- 172.21.15.151:0/2980858242 wait complete. 2023-09-30T02:02:59.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 Processor -- start 2023-09-30T02:02:59.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.374+0000 7f5338b2a700 1 -- start start 2023-09-30T02:02:59.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f533406b460 0x7f5334088c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5334149520 0x7f53341513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5334155200 con 0x7f533406b830 2023-09-30T02:02:59.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533414f010 con 0x7f533406b460 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5334154ec0 con 0x7f5334149520 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53432/0 (socket says 172.21.15.151:53432) 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 -- 172.21.15.151:0/2356464659 learned_addr learned my addr 172.21.15.151:0/2356464659 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5332d9d700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5334149520 0x7f53341513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f533259c700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f533406b460 0x7f5334088c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 -- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5334149520 msgr2=0x7f53341513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5334149520 0x7f53341513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 -- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f533406b460 msgr2=0x7f5334088c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f533406b460 0x7f5334088c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 -- 172.21.15.151:0/2356464659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009420 con 0x7f533406b830 2023-09-30T02:02:59.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5329d9b700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c810 tx=0x7f531c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c005c00 con 0x7f533406b830 2023-09-30T02:02:59.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c005d60 con 0x7f533406b830 2023-09-30T02:02:59.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c007320 con 0x7f533406b830 2023-09-30T02:02:59.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5334089960 con 0x7f533406b830 2023-09-30T02:02:59.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53341519b0 con 0x7f533406b830 2023-09-30T02:02:59.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.378+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5334155370 con 0x7f533406b830 2023-09-30T02:02:59.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.382+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f531c01b3b0 con 0x7f533406b830 2023-09-30T02:02:59.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.382+0000 7f532affd700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 0x7f5314062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.382+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f531c07ba20 con 0x7f533406b830 2023-09-30T02:02:59.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.382+0000 7f533259c700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 0x7f5314062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.382+0000 7f533259c700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 0x7f5314062a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53240093f0 tx=0x7f5324017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.390+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f531c04c200 con 0x7f533406b830 2023-09-30T02:02:59.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.546+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5334067110 con 0x7f5314060580 2023-09-30T02:02:59.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.550+0000 7f532affd700 1 -- 172.21.15.151:0/2356464659 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5334067110 con 0x7f5314060580 2023-09-30T02:02:59.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 msgr2=0x7f5314062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 0x7f5314062a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53240093f0 tx=0x7f5324017000 comp rx=0 tx=0).stop 2023-09-30T02:02:59.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 msgr2=0x7f5334154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c810 tx=0x7f531c007d90 comp rx=0 tx=0).stop 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 shutdown_connections 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f5314060580 0x7f5314062a30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5334149520 0x7f53341513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f533406b830 0x7f5334154980 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 --2- 172.21.15.151:0/2356464659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f533406b460 0x7f5334088c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 >> 172.21.15.151:0/2356464659 conn(0x7f5334070510 msgr2=0x7f533408a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 shutdown_connections 2023-09-30T02:02:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.554+0000 7f5338b2a700 1 -- 172.21.15.151:0/2356464659 wait complete. 2023-09-30T02:02:59.557 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T02:02:59.571 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T02:02:59.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:02:59.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 Processor -- start 2023-09-30T02:02:59.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 -- start start 2023-09-30T02:02:59.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f22100a91a0 0x7f22100aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2210140a40 0x7f2210140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22100af940 con 0x7f22100b0ec0 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22100aa820 con 0x7f2210140a40 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f221594a700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22100a95c0 con 0x7f22100a91a0 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f2215149700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2210140a40 0x7f2210140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f2215149700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2210140a40 0x7f2210140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53438/0 (socket says 172.21.15.151:53438) 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f2215149700 1 -- 172.21.15.151:0/1496008510 learned_addr learned my addr 172.21.15.151:0/1496008510 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.682+0000 7f220ffff700 1 --2- 172.21.15.151:0/1496008510 >> v2:172.21.15.172:3300/0 conn(0x7f22100a91a0 0x7f22100aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1496008510 >> v2:172.21.15.172:3300/0 conn(0x7f22100a91a0 msgr2=0x7f22100aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1496008510 >> v2:172.21.15.172:3300/0 conn(0x7f22100a91a0 0x7f22100aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1496008510 >> v1:172.21.15.172:6789/0 conn(0x7f22100b0ec0 legacy=0x7f22100ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1496008510 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100aad80 con 0x7f2210140a40 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220f7fe700 1 -- 172.21.15.151:0/1496008510 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886210818 0 0) 0x7f22100af940 con 0x7f22100b0ec0 2023-09-30T02:02:59.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1496008510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f2210140ea0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f221006abc0 tx=0x7f2208009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2d592445bbdbbe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220f7fe700 1 -- 172.21.15.151:0/1496008510 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800e070 con 0x7f2210140a40 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220f7fe700 1 -- 172.21.15.151:0/1496008510 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220800ba20 con 0x7f2210140a40 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220f7fe700 1 -- 172.21.15.151:0/1496008510 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080092e0 con 0x7f2210140a40 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- 172.21.15.151:0/1496008510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 msgr2=0x7f2210140ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- 172.21.15.151:0/1496008510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f2210140ea0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f221006abc0 tx=0x7f2208009840 comp rx=0 tx=0).stop 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- 172.21.15.151:0/1496008510 shutdown_connections 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- 172.21.15.151:0/1496008510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f2210140ea0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- 172.21.15.151:0/1496008510 >> v2:172.21.15.172:3300/0 conn(0x7f22100a91a0 0x7f22100aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- 172.21.15.151:0/1496008510 >> 172.21.15.151:0/1496008510 conn(0x7f22100b4670 msgr2=0x7f22100b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- 172.21.15.151:0/1496008510 shutdown_connections 2023-09-30T02:02:59.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- 172.21.15.151:0/1496008510 wait complete. 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 Processor -- start 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- start start 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22100a91a0 0x7f221006e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22100b0ec0 0x7f221006be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f221006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221011c070 con 0x7f2210140a40 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221011d320 con 0x7f22100a91a0 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f221594a700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221011d590 con 0x7f22100b0ec0 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22100b0ec0 0x7f221006be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220ffff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22100b0ec0 0x7f221006be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52572/0 (socket says 172.21.15.151:52572) 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f220ffff700 1 -- 172.21.15.151:0/1393734859 learned_addr learned my addr 172.21.15.151:0/1393734859 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f221006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2214948700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22100a91a0 0x7f221006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22100b0ec0 msgr2=0x7f221006be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f22100b0ec0 0x7f221006be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22100a91a0 msgr2=0x7f221006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22100a91a0 0x7f221006e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2208009ae0 con 0x7f2210140a40 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2214948700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f22100a91a0 0x7f221006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:02:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.686+0000 7f2215149700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f221006c3a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f22100afe90 tx=0x7f2208018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208008b70 con 0x7f2210140a40 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208008cf0 con 0x7f2210140a40 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221011bcb0 con 0x7f2210140a40 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080266b0 con 0x7f2210140a40 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221013b390 con 0x7f2210140a40 2023-09-30T02:02:59.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22100670a0 con 0x7f2210140a40 2023-09-30T02:02:59.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2208003040 con 0x7f2210140a40 2023-09-30T02:02:59.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.690+0000 7f220cff9700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f21fc060550 0x7f21fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f220807ea80 con 0x7f2210140a40 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f2214948700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f21fc060550 0x7f21fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220cff9700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7f21fc066920 0x7f21fc068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21fc0693e0 con 0x7f21fc066920 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f220807ed70 con 0x7f2210140a40 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220ffff700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7f21fc066920 0x7f21fc068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220ffff700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7f21fc066920 0x7f21fc068d30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f2214948700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f21fc060550 0x7f21fc062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22000011e0 tx=0x7f2200009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.694+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== osd.3 v2:172.21.15.151:6800/31360 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f21fc0693e0 con 0x7f21fc066920 2023-09-30T02:02:59.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.710+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 --> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f221013eef0 con 0x7f21fc066920 2023-09-30T02:02:59.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.710+0000 7f220cff9700 1 -- 172.21.15.151:0/1393734859 <== osd.3 v2:172.21.15.151:6800/31360 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f221013eef0 con 0x7f21fc066920 2023-09-30T02:02:59.714 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:02:59.714 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:02:59.714 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:02:59.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.710+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7f21fc066920 msgr2=0x7f21fc068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.710+0000 7f221594a700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:6800/31360,v1:172.21.15.151:6801/31360] conn(0x7f21fc066920 0x7f21fc068d30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f21fc060550 msgr2=0x7f21fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f21fc060550 0x7f21fc062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22000011e0 tx=0x7f2200009040 comp rx=0 tx=0).stop 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 msgr2=0x7f221006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 --2- 172.21.15.151:0/1393734859 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2210140a40 0x7f221006c3a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f22100afe90 tx=0x7f2208018000 comp rx=0 tx=0).stop 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f2215149700 1 -- 172.21.15.151:0/1393734859 reap_dead start 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 shutdown_connections 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 >> 172.21.15.151:0/1393734859 conn(0x7f22100b4670 msgr2=0x7f22100a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 shutdown_connections 2023-09-30T02:02:59.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.714+0000 7f221594a700 1 -- 172.21.15.151:0/1393734859 wait complete. 2023-09-30T02:02:59.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T02:02:59.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 Processor -- start 2023-09-30T02:02:59.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 -- start start 2023-09-30T02:02:59.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12240b46b0 0x7f122406db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122406d280 con 0x7f122406e0c0 2023-09-30T02:02:59.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122406cfa0 con 0x7f12240b42e0 2023-09-30T02:02:59.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1223fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122406b6f0 con 0x7f12240b46b0 2023-09-30T02:02:59.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12237fe700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f122406e0c0 0x7f1224140a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48092/0 (socket says 172.21.15.151:48092) 2023-09-30T02:02:59.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12237fe700 1 -- 172.21.15.151:0/2126923247 learned_addr learned my addr 172.21.15.151:0/2126923247 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:02:59.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12227fc700 1 --2- 172.21.15.151:0/2126923247 >> v2:172.21.15.172:3300/0 conn(0x7f12240b46b0 0x7f122406db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855128360 0 0) 0x7f122406d280 con 0x7f122406e0c0 2023-09-30T02:02:59.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120c003570 con 0x7f122406e0c0 2023-09-30T02:02:59.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488168387 0 0) 0x7f122406cfa0 con 0x7f12240b42e0 2023-09-30T02:02:59.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122406d280 con 0x7f12240b42e0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12227fc700 1 -- 172.21.15.151:0/2126923247 >> v1:172.21.15.172:6789/0 conn(0x7f12240b42e0 legacy=0x7f122406af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337639418 0 0) 0x7f120c003570 con 0x7f122406e0c0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12227fc700 1 -- 172.21.15.151:0/2126923247 >> v1:172.21.15.151:6789/0 conn(0x7f122406e0c0 legacy=0x7f1224140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12227fc700 1 -- 172.21.15.151:0/2126923247 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122406e560 con 0x7f12240b46b0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f12227fc700 1 --2- 172.21.15.151:0/2126923247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12240b46b0 0x7f122406db80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f12180098e0 tx=0x7f12180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5c5875fff8b00c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c001470 con 0x7f12240b46b0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1218011420 con 0x7f12240b46b0 2023-09-30T02:02:59.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.834+0000 7f1221ffb700 1 -- 172.21.15.151:0/2126923247 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180117c0 con 0x7f12240b46b0 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- 172.21.15.151:0/2126923247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12240b46b0 msgr2=0x7f122406db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 --2- 172.21.15.151:0/2126923247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12240b46b0 0x7f122406db80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f12180098e0 tx=0x7f12180091a0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- 172.21.15.151:0/2126923247 shutdown_connections 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 --2- 172.21.15.151:0/2126923247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12240b46b0 0x7f122406db80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- 172.21.15.151:0/2126923247 >> 172.21.15.151:0/2126923247 conn(0x7f1224070f90 msgr2=0x7f12240b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- 172.21.15.151:0/2126923247 shutdown_connections 2023-09-30T02:02:59.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- 172.21.15.151:0/2126923247 wait complete. 2023-09-30T02:02:59.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 Processor -- start 2023-09-30T02:02:59.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- start start 2023-09-30T02:02:59.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f122406e0c0 0x7f122414cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 0x7f12240aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240ac2b0 con 0x7f12240b46b0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240abf70 con 0x7f122406e0c0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1223fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122413c810 con 0x7f12240b42e0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12227fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 0x7f12240aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53452/0 (socket says 172.21.15.151:53452) 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 -- 172.21.15.151:0/352699418 learned_addr learned my addr 172.21.15.151:0/352699418 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f1222ffd700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f122406e0c0 0x7f122414cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12227fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 0x7f12240aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52582/0 (socket says 172.21.15.151:52582) 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 -- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 msgr2=0x7f12240aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 0x7f12240aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 -- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f122406e0c0 msgr2=0x7f122414cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.838+0000 7f12237fe700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f122406e0c0 0x7f122414cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f12237fe700 1 -- 172.21.15.151:0/352699418 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218009420 con 0x7f12240b46b0 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f12237fe700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f121c0011e0 tx=0x7f121c0019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c007360 con 0x7f12240b46b0 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c0074c0 con 0x7f12240b46b0 2023-09-30T02:02:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12240aadf0 con 0x7f12240b46b0 2023-09-30T02:02:59.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c004830 con 0x7f12240b46b0 2023-09-30T02:02:59.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12240ac8e0 con 0x7f12240b46b0 2023-09-30T02:02:59.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12240ac460 con 0x7f12240b46b0 2023-09-30T02:02:59.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f121c004be0 con 0x7f12240b46b0 2023-09-30T02:02:59.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 0x7f1210062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:02:59.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f121c07bb10 con 0x7f12240b46b0 2023-09-30T02:02:59.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f1222ffd700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 0x7f1210062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:02:59.851 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.842+0000 7f1222ffd700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 0x7f1210062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1214004140 tx=0x7f1214007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:02:59.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:02:59.854+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f12240ac460 con 0x7f12240b46b0 2023-09-30T02:03:00.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.010+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f12240ac460 con 0x7f1210060550 2023-09-30T02:03:00.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.010+0000 7f120affd700 1 -- 172.21.15.151:0/352699418 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f12240ac460 con 0x7f1210060550 2023-09-30T02:03:00.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 msgr2=0x7f1210062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 0x7f1210062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1214004140 tx=0x7f1214007f90 comp rx=0 tx=0).stop 2023-09-30T02:03:00.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 msgr2=0x7f12240aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f121c0011e0 tx=0x7f121c0019e0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 shutdown_connections 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f1210060550 0x7f1210062a00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12240b46b0 0x7f12240aba30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12240b42e0 0x7f12240aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 --2- 172.21.15.151:0/352699418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f122406e0c0 0x7f122414cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 >> 172.21.15.151:0/352699418 conn(0x7f1224070f90 msgr2=0x7f12240b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 shutdown_connections 2023-09-30T02:03:00.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.014+0000 7f1223fff700 1 -- 172.21.15.151:0/352699418 wait complete. 2023-09-30T02:03:00.017 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T02:03:00.026 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T02:03:00.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:03:00.137 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.134+0000 7f3147fff700 1 Processor -- start 2023-09-30T02:03:00.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.134+0000 7f3147fff700 1 -- start start 2023-09-30T02:03:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31480abf50 con 0x7f314813d850 2023-09-30T02:03:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31480a9540 con 0x7f31480a9f20 2023-09-30T02:03:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314806bd50 con 0x7f3148070740 2023-09-30T02:03:00.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52592/0 (socket says 172.21.15.151:52592) 2023-09-30T02:03:00.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 -- 172.21.15.151:0/2191636181 learned_addr learned my addr 172.21.15.151:0/2191636181 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 -- 172.21.15.151:0/2191636181 >> v1:172.21.15.172:6789/0 conn(0x7f31480a9f20 legacy=0x7f314806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 -- 172.21.15.151:0/2191636181 >> v1:172.21.15.151:6789/0 conn(0x7f314813d850 legacy=0x7f314813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 -- 172.21.15.151:0/2191636181 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148070b10 con 0x7f3148070740 2023-09-30T02:03:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3145ffb700 1 -- 172.21.15.151:0/2191636181 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191109359 0 0) 0x7f31480abf50 con 0x7f314813d850 2023-09-30T02:03:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3146ffd700 1 --2- 172.21.15.151:0/2191636181 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f313800bc80 tx=0x7f313800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=325223bc940880e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3145ffb700 1 -- 172.21.15.151:0/2191636181 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800be90 con 0x7f3148070740 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3145ffb700 1 -- 172.21.15.151:0/2191636181 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313800ddc0 con 0x7f3148070740 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3145ffb700 1 -- 172.21.15.151:0/2191636181 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31380106a0 con 0x7f3148070740 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- 172.21.15.151:0/2191636181 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 msgr2=0x7f31480ac3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 --2- 172.21.15.151:0/2191636181 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f313800bc80 tx=0x7f313800b4f0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- 172.21.15.151:0/2191636181 shutdown_connections 2023-09-30T02:03:00.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 --2- 172.21.15.151:0/2191636181 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f31480ac3e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- 172.21.15.151:0/2191636181 >> 172.21.15.151:0/2191636181 conn(0x7f3148067370 msgr2=0x7f3148067770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- 172.21.15.151:0/2191636181 shutdown_connections 2023-09-30T02:03:00.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.138+0000 7f3147fff700 1 -- 172.21.15.151:0/2191636181 wait complete. 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 Processor -- start 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- start start 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f314814c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31480a9f20 0x7f314814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f314813d850 0x7f3148097be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314809bac0 con 0x7f31480a9f20 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314814cff0 con 0x7f314813d850 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31480a6750 con 0x7f3148070740 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3146ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f314814c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31467fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31480a9f20 0x7f314814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3146ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f314814c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52604/0 (socket says 172.21.15.151:52604) 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3146ffd700 1 -- 172.21.15.151:0/2887758471 learned_addr learned my addr 172.21.15.151:0/2887758471 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f314813d850 0x7f3148097be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 -- 172.21.15.151:0/2887758471 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 msgr2=0x7f314814c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3148070740 0x7f314814c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 -- 172.21.15.151:0/2887758471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31480a9f20 msgr2=0x7f314814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31480a9f20 0x7f314814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313800b790 con 0x7f314813d850 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31467fc700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31480a9f20 0x7f314814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f31477fe700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f314813d850 0x7f3148097be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f314000c7e0 tx=0x7f31400015f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140012630 con 0x7f314813d850 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31400127b0 con 0x7f314813d850 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3148099880 con 0x7f314813d850 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140012b70 con 0x7f314813d850 2023-09-30T02:03:00.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.142+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3148098390 con 0x7f314813d850 2023-09-30T02:03:00.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31480670a0 con 0x7f314813d850 2023-09-30T02:03:00.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3140003030 con 0x7f314813d850 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3134060550 0x7f3134062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f314007c670 con 0x7f314813d850 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f3146ffd700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3134060550 0x7f3134062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f3134066920 0x7f3134068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31340693e0 con 0x7f3134066920 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f314007c960 con 0x7f314813d850 2023-09-30T02:03:00.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.146+0000 7f31467fc700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f3134066920 0x7f3134068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.150+0000 7f3146ffd700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3134060550 0x7f3134062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f313800b760 tx=0x7f313801a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.150+0000 7f31467fc700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f3134066920 0x7f3134068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.152 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.150+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== osd.4 v2:172.21.15.172:6800/31451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f31340693e0 con 0x7f3134066920 2023-09-30T02:03:00.171 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 --> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f314814d130 con 0x7f3134066920 2023-09-30T02:03:00.171 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f312effd700 1 -- 172.21.15.151:0/2887758471 <== osd.4 v2:172.21.15.172:6800/31451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f314814d130 con 0x7f3134066920 2023-09-30T02:03:00.172 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:03:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:03:00.172 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:03:00.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f3134066920 msgr2=0x7f3134068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6800/31451,v1:172.21.15.172:6801/31451] conn(0x7f3134066920 0x7f3134068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.173 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3134060550 msgr2=0x7f3134062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f3134060550 0x7f3134062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f313800b760 tx=0x7f313801a000 comp rx=0 tx=0).stop 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f314813d850 msgr2=0x7f3148097be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 --2- 172.21.15.151:0/2887758471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f314813d850 0x7f3148097be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f314000c7e0 tx=0x7f31400015f0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f31477fe700 1 -- 172.21.15.151:0/2887758471 reap_dead start 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 shutdown_connections 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 >> 172.21.15.151:0/2887758471 conn(0x7f3148067370 msgr2=0x7f31480a70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 shutdown_connections 2023-09-30T02:03:00.174 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.170+0000 7f3147fff700 1 -- 172.21.15.151:0/2887758471 wait complete. 2023-09-30T02:03:00.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T02:03:00.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.302+0000 7f8e9eca5700 1 Processor -- start 2023-09-30T02:03:00.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.302+0000 7f8e9eca5700 1 -- start start 2023-09-30T02:03:00.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.302+0000 7f8e9eca5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e9806ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8e980ad1e0 0x7f8e9813a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9806ca00 con 0x7f8e9806ec40 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e98138090 con 0x7f8e980ad1e0 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9806e3c0 con 0x7f8e9806e870 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8e980ad1e0 0x7f8e9813a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9ca41700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e9806ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8e980ad1e0 0x7f8e9813a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:53460/0 (socket says 172.21.15.151:53460) 2023-09-30T02:03:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 -- 172.21.15.151:0/3892605929 learned_addr learned my addr 172.21.15.151:0/3892605929 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 -- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 msgr2=0x7f8e9806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 --2- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e9806ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 -- 172.21.15.151:0/3892605929 >> v1:172.21.15.172:6789/0 conn(0x7f8e9806ec40 legacy=0x7f8e980a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 -- 172.21.15.151:0/3892605929 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9806c6d0 con 0x7f8e980ad1e0 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e977fe700 1 -- 172.21.15.151:0/3892605929 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429966002 0 0) 0x7f8e9806ca00 con 0x7f8e9806ec40 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e97fff700 1 --2- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e9813a400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009f20 tx=0x7f8e80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe9d1d5dff9996f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e977fe700 1 -- 172.21.15.151:0/3892605929 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8000e070 con 0x7f8e980ad1e0 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e977fe700 1 -- 172.21.15.151:0/3892605929 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8000b9d0 con 0x7f8e980ad1e0 2023-09-30T02:03:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e977fe700 1 -- 172.21.15.151:0/3892605929 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e800092e0 con 0x7f8e980ad1e0 2023-09-30T02:03:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 msgr2=0x7f8e9813a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e9813a400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009f20 tx=0x7f8e80009840 comp rx=0 tx=0).stop 2023-09-30T02:03:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- 172.21.15.151:0/3892605929 shutdown_connections 2023-09-30T02:03:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e9813a400 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/3892605929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e9806ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- 172.21.15.151:0/3892605929 >> 172.21.15.151:0/3892605929 conn(0x7f8e980708b0 msgr2=0x7f8e98070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- 172.21.15.151:0/3892605929 shutdown_connections 2023-09-30T02:03:00.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.306+0000 7f8e9eca5700 1 -- 172.21.15.151:0/3892605929 wait complete. 2023-09-30T02:03:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 Processor -- start 2023-09-30T02:03:00.312 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 -- start start 2023-09-30T02:03:00.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e9806ec40 0x7f8e9814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e980a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980a4790 con 0x7f8e980ad1e0 2023-09-30T02:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9814d7f0 con 0x7f8e9806ec40 2023-09-30T02:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9eca5700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9814e250 con 0x7f8e9806e870 2023-09-30T02:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9ca41700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9ca41700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:52622/0 (socket says 172.21.15.151:52622) 2023-09-30T02:03:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9ca41700 1 -- 172.21.15.151:0/2299217291 learned_addr learned my addr 172.21.15.151:0/2299217291 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e9d242700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e980a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.310+0000 7f8e97fff700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e9806ec40 0x7f8e9814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 -- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e9806ec40 msgr2=0x7f8e9814dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e9806ec40 0x7f8e9814dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 -- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 msgr2=0x7f8e980a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e980a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 -- 172.21.15.151:0/2299217291 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80009ae0 con 0x7f8e9806e870 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9d242700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e980a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9ca41700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e88010e90 tx=0x7f8e880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800edf0 con 0x7f8e9806e870 2023-09-30T02:03:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e88011450 con 0x7f8e9806e870 2023-09-30T02:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9814e470 con 0x7f8e9806e870 2023-09-30T02:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880117f0 con 0x7f8e9806e870 2023-09-30T02:03:00.320 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e980a6890 con 0x7f8e9806e870 2023-09-30T02:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9814d9b0 con 0x7f8e9806e870 2023-09-30T02:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8e88011ba0 con 0x7f8e9806e870 2023-09-30T02:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.314+0000 7f8e94ff9700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 0x7f8e78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.318+0000 7f8e97fff700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 0x7f8e78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.318+0000 7f8e97fff700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 0x7f8e78062a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009880 tx=0x7f8e80018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.318+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8e88053420 con 0x7f8e9806e870 2023-09-30T02:03:00.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.326+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e8804f020 con 0x7f8e9806e870 2023-09-30T02:03:00.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e980a6890 con 0x7f8e78060550 2023-09-30T02:03:00.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e94ff9700 1 -- 172.21.15.151:0/2299217291 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e980a6890 con 0x7f8e78060550 2023-09-30T02:03:00.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 msgr2=0x7f8e78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 0x7f8e78062a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009880 tx=0x7f8e80018000 comp rx=0 tx=0).stop 2023-09-30T02:03:00.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 msgr2=0x7f8e980a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e88010e90 tx=0x7f8e880091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 shutdown_connections 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f8e78060550 0x7f8e78062a00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e980ad1e0 0x7f8e980a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e9806ec40 0x7f8e9814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 --2- 172.21.15.151:0/2299217291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e9806e870 0x7f8e980a7da0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.482+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 >> 172.21.15.151:0/2299217291 conn(0x7f8e980708b0 msgr2=0x7f8e98070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.486+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 shutdown_connections 2023-09-30T02:03:00.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.486+0000 7f8e9eca5700 1 -- 172.21.15.151:0/2299217291 wait complete. 2023-09-30T02:03:00.488 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T02:03:00.497 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T02:03:00.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:03:00.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.602+0000 7f2b9e147700 1 Processor -- start 2023-09-30T02:03:00.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.602+0000 7f2b9e147700 1 -- start start 2023-09-30T02:03:00.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980b0ec0 0x7f2b980a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2b980ab360 0x7f2b980ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980af940 con 0x7f2b980abcb0 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980aa820 con 0x7f2b980ab360 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980aad80 con 0x7f2b980b0ec0 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2b980ab360 0x7f2b980ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9d145700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980b0ec0 0x7f2b980a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2b980ab360 0x7f2b980ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45000/0 (socket says 172.21.15.151:45000) 2023-09-30T02:03:00.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 -- 172.21.15.151:0/3687061529 learned_addr learned my addr 172.21.15.151:0/3687061529 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 -- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980b0ec0 msgr2=0x7f2b980a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:00.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 --2- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980b0ec0 0x7f2b980a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 -- 172.21.15.151:0/3687061529 >> v1:172.21.15.172:6789/0 conn(0x7f2b980abcb0 legacy=0x7f2b980ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 -- 172.21.15.151:0/3687061529 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980a9870 con 0x7f2b980ab360 2023-09-30T02:03:00.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9c944700 1 --2- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980ab360 0x7f2b980ab770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0099b0 tx=0x7f2b8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb95f745d70f597 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b8bfff700 1 -- 172.21.15.151:0/3687061529 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c009ea0 con 0x7f2b980ab360 2023-09-30T02:03:00.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b8bfff700 1 -- 172.21.15.151:0/3687061529 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c00b860 con 0x7f2b980ab360 2023-09-30T02:03:00.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b8bfff700 1 -- 172.21.15.151:0/3687061529 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00bc00 con 0x7f2b980ab360 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980ab360 msgr2=0x7f2b980ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 --2- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980ab360 0x7f2b980ab770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0099b0 tx=0x7f2b8c009250 comp rx=0 tx=0).stop 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- 172.21.15.151:0/3687061529 shutdown_connections 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 --2- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980ab360 0x7f2b980ab770 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 --2- 172.21.15.151:0/3687061529 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980b0ec0 0x7f2b980a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- 172.21.15.151:0/3687061529 >> 172.21.15.151:0/3687061529 conn(0x7f2b980b4670 msgr2=0x7f2b980b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- 172.21.15.151:0/3687061529 shutdown_connections 2023-09-30T02:03:00.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.606+0000 7f2b9e147700 1 -- 172.21.15.151:0/3687061529 wait complete. 2023-09-30T02:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 Processor -- start 2023-09-30T02:03:00.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 -- start start 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980ab360 0x7f2b9811ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980abcb0 0x7f2b9811eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 0x7f2b9814d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9811f0d0 con 0x7f2b980b0ec0 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9811ff90 con 0x7f2b980ab360 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9e147700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98120200 con 0x7f2b980abcb0 2023-09-30T02:03:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9d946700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 0x7f2b9814d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9c944700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980abcb0 0x7f2b9811eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9c944700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980abcb0 0x7f2b9811eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56432/0 (socket says 172.21.15.151:56432) 2023-09-30T02:03:00.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9d946700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 0x7f2b9814d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45016/0 (socket says 172.21.15.151:45016) 2023-09-30T02:03:00.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9c944700 1 -- 172.21.15.151:0/4098438571 learned_addr learned my addr 172.21.15.151:0/4098438571 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.610+0000 7f2b9d145700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980ab360 0x7f2b9811ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 -- 172.21.15.151:0/4098438571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980abcb0 msgr2=0x7f2b9811eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2b980abcb0 0x7f2b9811eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 -- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980ab360 msgr2=0x7f2b9811ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980ab360 0x7f2b9811ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0094f0 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9d946700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 0x7f2b9814d380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c860 tx=0x7f2b90001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90012630 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b90012790 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9814ce60 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90012b50 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9811ba40 con 0x7f2b980b0ec0 2023-09-30T02:03:00.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b980670a0 con 0x7f2b980b0ec0 2023-09-30T02:03:00.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2b90024430 con 0x7f2b980b0ec0 2023-09-30T02:03:00.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b7c060550 0x7f2b7c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.614+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2b9007cea0 con 0x7f2b980b0ec0 2023-09-30T02:03:00.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b897fa700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f2b7c066920 0x7f2b7c068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b7c0693e0 con 0x7f2b7c066920 2023-09-30T02:03:00.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b9007d270 con 0x7f2b980b0ec0 2023-09-30T02:03:00.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b9d145700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b7c060550 0x7f2b7c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b9c944700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f2b7c066920 0x7f2b7c068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b9d145700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b7c060550 0x7f2b7c062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009db0 tx=0x7f2b940091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b9c944700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f2b7c066920 0x7f2b7c068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.618+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== osd.5 v2:172.21.15.172:6810/31452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2b7c0693e0 con 0x7f2b7c066920 2023-09-30T02:03:00.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.630+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 --> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2b980b1290 con 0x7f2b7c066920 2023-09-30T02:03:00.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b897fa700 1 -- 172.21.15.151:0/4098438571 <== osd.5 v2:172.21.15.172:6810/31452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2b980b1290 con 0x7f2b7c066920 2023-09-30T02:03:00.635 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:03:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:03:00.635 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f2b7c066920 msgr2=0x7f2b7c068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6810/31452,v1:172.21.15.172:6811/31452] conn(0x7f2b7c066920 0x7f2b7c068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b7c060550 msgr2=0x7f2b7c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2b7c060550 0x7f2b7c062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009db0 tx=0x7f2b940091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 msgr2=0x7f2b9814d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 --2- 172.21.15.151:0/4098438571 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2b980b0ec0 0x7f2b9814d380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c860 tx=0x7f2b90001690 comp rx=0 tx=0).stop 2023-09-30T02:03:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9d946700 1 -- 172.21.15.151:0/4098438571 reap_dead start 2023-09-30T02:03:00.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 shutdown_connections 2023-09-30T02:03:00.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 >> 172.21.15.151:0/4098438571 conn(0x7f2b980b4670 msgr2=0x7f2b980ae410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 shutdown_connections 2023-09-30T02:03:00.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.634+0000 7f2b9e147700 1 -- 172.21.15.151:0/4098438571 wait complete. 2023-09-30T02:03:00.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T02:03:00.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 Processor -- start 2023-09-30T02:03:00.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- start start 2023-09-30T02:03:00.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 0x7f0a4813e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a480abe00 con 0x7f0a4806b550 2023-09-30T02:03:00.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4806c700 con 0x7f0a4813e4e0 2023-09-30T02:03:00.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a480ac820 con 0x7f0a4806b180 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56448/0 (socket says 172.21.15.151:56448) 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 -- 172.21.15.151:0/244930692 learned_addr learned my addr 172.21.15.151:0/244930692 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a477fe700 1 --2- 172.21.15.151:0/244930692 >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 0x7f0a4813e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 -- 172.21.15.151:0/244930692 >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 msgr2=0x7f0a4813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 --2- 172.21.15.151:0/244930692 >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 0x7f0a4813e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 -- 172.21.15.151:0/244930692 >> v1:172.21.15.172:6789/0 conn(0x7f0a4806b550 legacy=0x7f0a481415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 -- 172.21.15.151:0/244930692 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4806c8e0 con 0x7f0a4806b180 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a477fe700 1 --2- 172.21.15.151:0/244930692 >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 0x7f0a4813e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:00.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a46ffd700 1 --2- 172.21.15.151:0/244930692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a300099b0 tx=0x7f0a30009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4187afd5e0345e07 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a45ffb700 1 -- 172.21.15.151:0/244930692 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000b860 con 0x7f0a4806b180 2023-09-30T02:03:00.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a45ffb700 1 -- 172.21.15.151:0/244930692 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3000b9c0 con 0x7f0a4806b180 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a45ffb700 1 -- 172.21.15.151:0/244930692 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000bd60 con 0x7f0a4806b180 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- 172.21.15.151:0/244930692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 msgr2=0x7f0a480ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 --2- 172.21.15.151:0/244930692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a300099b0 tx=0x7f0a30009250 comp rx=0 tx=0).stop 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- 172.21.15.151:0/244930692 shutdown_connections 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 --2- 172.21.15.151:0/244930692 >> v2:172.21.15.172:3300/0 conn(0x7f0a4813e4e0 0x7f0a4813e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 --2- 172.21.15.151:0/244930692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4806b180 0x7f0a480ac2e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- 172.21.15.151:0/244930692 >> 172.21.15.151:0/244930692 conn(0x7f0a48070510 msgr2=0x7f0a48070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- 172.21.15.151:0/244930692 shutdown_connections 2023-09-30T02:03:00.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.754+0000 7f0a47fff700 1 -- 172.21.15.151:0/244930692 wait complete. 2023-09-30T02:03:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 Processor -- start 2023-09-30T02:03:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- start start 2023-09-30T02:03:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a4806b180 0x7f0a4806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4813e4e0 0x7f0a48151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481455f0 con 0x7f0a4806b550 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a48140770 con 0x7f0a4806b180 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481549c0 con 0x7f0a4813e4e0 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a477fe700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4813e4e0 0x7f0a48151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45026/0 (socket says 172.21.15.151:45026) 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 -- 172.21.15.151:0/3481593400 learned_addr learned my addr 172.21.15.151:0/3481593400 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a46ffd700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a4806b180 0x7f0a4806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 -- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4813e4e0 msgr2=0x7f0a48151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4813e4e0 0x7f0a48151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 -- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a4806b180 msgr2=0x7f0a4806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a4806b180 0x7f0a4806e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 -- 172.21.15.151:0/3481593400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300094f0 con 0x7f0a4806b550 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a467fc700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000cb10 tx=0x7f0a400017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4000ce20 con 0x7f0a4806b550 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4000a7e0 con 0x7f0a4806b550 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4000aba0 con 0x7f0a4806b550 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a481413b0 con 0x7f0a4806b550 2023-09-30T02:03:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.758+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a48145220 con 0x7f0a4806b550 2023-09-30T02:03:00.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a480008d0 con 0x7f0a4806b550 2023-09-30T02:03:00.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0a40003030 con 0x7f0a4806b550 2023-09-30T02:03:00.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a3effd700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 0x7f0a28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:00.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0a4007c590 con 0x7f0a4806b550 2023-09-30T02:03:00.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a46ffd700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 0x7f0a28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:00.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.762+0000 7f0a46ffd700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 0x7f0a28062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a300090b0 tx=0x7f0a30017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:00.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.770+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a4004cd70 con 0x7f0a4806b550 2023-09-30T02:03:00.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.926+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a48154b80 con 0x7f0a28060550 2023-09-30T02:03:00.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a3effd700 1 -- 172.21.15.151:0/3481593400 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0a48154b80 con 0x7f0a28060550 2023-09-30T02:03:00.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 msgr2=0x7f0a28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 0x7f0a28062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a300090b0 tx=0x7f0a30017000 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 msgr2=0x7f0a48143140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000cb10 tx=0x7f0a400017b0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 shutdown_connections 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f0a28060550 0x7f0a28062a00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a4813e4e0 0x7f0a48151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a4806b550 0x7f0a48143140 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 --2- 172.21.15.151:0/3481593400 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a4806b180 0x7f0a4806e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 >> 172.21.15.151:0/3481593400 conn(0x7f0a48070510 msgr2=0x7f0a4813aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:00.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.930+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 shutdown_connections 2023-09-30T02:03:00.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:00.934+0000 7f0a47fff700 1 -- 172.21.15.151:0/3481593400 wait complete. 2023-09-30T02:03:00.936 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T02:03:00.946 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T02:03:00.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:03:01.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 Processor -- start 2023-09-30T02:03:01.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 -- start start 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78880a4a40 0x7f78880a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78880abed0 con 0x7f788813d360 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78880a9490 con 0x7f78880a4670 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e827700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880a5f00 con 0x7f78880a4a40 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e026700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f788813d360 0x7f788813d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48774/0 (socket says 172.21.15.151:48774) 2023-09-30T02:03:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788e026700 1 -- 172.21.15.151:0/3227390052 learned_addr learned my addr 172.21.15.151:0/3227390052 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:03:01.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788d024700 1 --2- 172.21.15.151:0/3227390052 >> v2:172.21.15.172:3300/0 conn(0x7f78880a4a40 0x7f78880a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788c823700 1 -- 172.21.15.151:0/3227390052 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032264664 0 0) 0x7f78880abed0 con 0x7f788813d360 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788c823700 1 -- 172.21.15.151:0/3227390052 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7870003570 con 0x7f788813d360 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788d024700 1 -- 172.21.15.151:0/3227390052 >> v1:172.21.15.172:6789/0 conn(0x7f78880a4670 legacy=0x7f78880a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788d024700 1 -- 172.21.15.151:0/3227390052 >> v1:172.21.15.151:6789/0 conn(0x7f788813d360 legacy=0x7f788813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788d024700 1 -- 172.21.15.151:0/3227390052 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78880a61e0 con 0x7f78880a4a40 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.058+0000 7f788d024700 1 --2- 172.21.15.151:0/3227390052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4a40 0x7f78880a8f50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f78800098d0 tx=0x7f7880009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f2c85752b2acaa1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788c823700 1 -- 172.21.15.151:0/3227390052 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880005c10 con 0x7f78880a4a40 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788c823700 1 -- 172.21.15.151:0/3227390052 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880005d70 con 0x7f78880a4a40 2023-09-30T02:03:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788c823700 1 -- 172.21.15.151:0/3227390052 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880006280 con 0x7f78880a4a40 2023-09-30T02:03:01.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 -- 172.21.15.151:0/3227390052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4a40 msgr2=0x7f78880a8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 --2- 172.21.15.151:0/3227390052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4a40 0x7f78880a8f50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f78800098d0 tx=0x7f7880009120 comp rx=0 tx=0).stop 2023-09-30T02:03:01.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 -- 172.21.15.151:0/3227390052 shutdown_connections 2023-09-30T02:03:01.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 --2- 172.21.15.151:0/3227390052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4a40 0x7f78880a8f50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 -- 172.21.15.151:0/3227390052 >> 172.21.15.151:0/3227390052 conn(0x7f7888070fe0 msgr2=0x7f78880af770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 -- 172.21.15.151:0/3227390052 shutdown_connections 2023-09-30T02:03:01.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.062+0000 7f788e827700 1 -- 172.21.15.151:0/3227390052 wait complete. 2023-09-30T02:03:01.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 Processor -- start 2023-09-30T02:03:01.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- start start 2023-09-30T02:03:01.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4670 0x7f788813a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78880a4a40 0x7f7888135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 0x7f78881361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888140450 con 0x7f788813d360 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888136720 con 0x7f78880a4670 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788813eaf0 con 0x7f78880a4a40 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 0x7f78881361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 0x7f78881361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45032/0 (socket says 172.21.15.151:45032) 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788d024700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78880a4a40 0x7f7888135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 -- 172.21.15.151:0/3391508902 learned_addr learned my addr 172.21.15.151:0/3391508902 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788d825700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4670 0x7f788813a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 -- 172.21.15.151:0/3391508902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78880a4a40 msgr2=0x7f7888135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78880a4a40 0x7f7888135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 -- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4670 msgr2=0x7f788813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78880a4670 0x7f788813a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78800093c0 con 0x7f788813d360 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e026700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 0x7f78881361e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7878001460 tx=0x7f7878001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800a8a0 con 0x7f788813d360 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787800aa00 con 0x7f788813d360 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7888139550 con 0x7f788813d360 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800ad80 con 0x7f788813d360 2023-09-30T02:03:01.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788813e540 con 0x7f788813d360 2023-09-30T02:03:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.066+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78880670a0 con 0x7f788813d360 2023-09-30T02:03:01.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7878007300 con 0x7f788813d360 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7874060550 0x7f7874062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f787807c8f0 con 0x7f788813d360 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f7874066920 0x7f7874068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78740693e0 con 0x7f7874066920 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7878081240 con 0x7f788813d360 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f788d825700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7874060550 0x7f7874062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f788d024700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f7874066920 0x7f7874068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f788d825700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7874060550 0x7f7874062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f788d024700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f7874066920 0x7f7874068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.070+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== osd.6 v2:172.21.15.172:6824/31453 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f78740693e0 con 0x7f7874066920 2023-09-30T02:03:01.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.090+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 --> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f788813ded0 con 0x7f7874066920 2023-09-30T02:03:01.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f7885ffb700 1 -- 172.21.15.151:0/3391508902 <== osd.6 v2:172.21.15.172:6824/31453 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f788813ded0 con 0x7f7874066920 2023-09-30T02:03:01.094 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:03:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:03:01.095 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f7874066920 msgr2=0x7f7874068d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6824/31453,v1:172.21.15.172:6825/31453] conn(0x7f7874066920 0x7f7874068d30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7874060550 msgr2=0x7f7874062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f7874060550 0x7f7874062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c009040 comp rx=0 tx=0).stop 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 msgr2=0x7f78881361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 --2- 172.21.15.151:0/3391508902 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f788813d360 0x7f78881361e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7878001460 tx=0x7f7878001a40 comp rx=0 tx=0).stop 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e026700 1 -- 172.21.15.151:0/3391508902 reap_dead start 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 shutdown_connections 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 >> 172.21.15.151:0/3391508902 conn(0x7f7888070fe0 msgr2=0x7f788813ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 shutdown_connections 2023-09-30T02:03:01.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.094+0000 7f788e827700 1 -- 172.21.15.151:0/3391508902 wait complete. 2023-09-30T02:03:01.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T02:03:01.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 Processor -- start 2023-09-30T02:03:01.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 -- start start 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 0x7f2dac06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2dac0ad160 0x7f2dac136a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac1396c0 con 0x7f2dac069f70 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac06d460 con 0x7f2dac0ad160 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db3544700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac13a5e0 con 0x7f2dac0acd90 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db1ae1700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f2dac069f70 0x7f2dac13cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48782/0 (socket says 172.21.15.151:48782) 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db1ae1700 1 -- 172.21.15.151:0/78918905 learned_addr learned my addr 172.21.15.151:0/78918905 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 --2- 172.21.15.151:0/78918905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 0x7f2dac06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db0adf700 1 --2- 172.21.15.151:0/78918905 >> v2:172.21.15.172:3300/0 conn(0x7f2dac0ad160 0x7f2dac136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2dabfff700 1 -- 172.21.15.151:0/78918905 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470785033 0 0) 0x7f2dac1396c0 con 0x7f2dac069f70 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2dabfff700 1 -- 172.21.15.151:0/78918905 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003570 con 0x7f2dac069f70 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 -- 172.21.15.151:0/78918905 >> v2:172.21.15.172:3300/0 conn(0x7f2dac0ad160 msgr2=0x7f2dac136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 --2- 172.21.15.151:0/78918905 >> v2:172.21.15.172:3300/0 conn(0x7f2dac0ad160 0x7f2dac136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 -- 172.21.15.151:0/78918905 >> v1:172.21.15.151:6789/0 conn(0x7f2dac069f70 legacy=0x7f2dac13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 -- 172.21.15.151:0/78918905 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac06e580 con 0x7f2dac0acd90 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2db12e0700 1 --2- 172.21.15.151:0/78918905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 0x7f2dac06e040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2da00098e0 tx=0x7f2da00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78984e5f4c234a0d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2dabfff700 1 -- 172.21.15.151:0/78918905 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0011630 con 0x7f2dac0acd90 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2dabfff700 1 -- 172.21.15.151:0/78918905 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da0011790 con 0x7f2dac0acd90 2023-09-30T02:03:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.210+0000 7f2dabfff700 1 -- 172.21.15.151:0/78918905 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0011b10 con 0x7f2dac0acd90 2023-09-30T02:03:01.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- 172.21.15.151:0/78918905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 msgr2=0x7f2dac06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 --2- 172.21.15.151:0/78918905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 0x7f2dac06e040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2da00098e0 tx=0x7f2da00091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- 172.21.15.151:0/78918905 shutdown_connections 2023-09-30T02:03:01.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 --2- 172.21.15.151:0/78918905 >> v2:172.21.15.172:3300/0 conn(0x7f2dac0ad160 0x7f2dac136a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.216 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 --2- 172.21.15.151:0/78918905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac0acd90 0x7f2dac06e040 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- 172.21.15.151:0/78918905 >> 172.21.15.151:0/78918905 conn(0x7f2dac0708b0 msgr2=0x7f2dac070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- 172.21.15.151:0/78918905 shutdown_connections 2023-09-30T02:03:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- 172.21.15.151:0/78918905 wait complete. 2023-09-30T02:03:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 Processor -- start 2023-09-30T02:03:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.214+0000 7f2db3544700 1 -- start start 2023-09-30T02:03:01.220 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 0x7f2dac1376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dac0ad160 0x7f2dac12ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1389e0 con 0x7f2dac0acd90 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1392a0 con 0x7f2dac0ad160 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac138010 con 0x7f2dac069f70 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db0adf700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 0x7f2dac1376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56486/0 (socket says 172.21.15.151:56486) 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 -- 172.21.15.151:0/1754354633 learned_addr learned my addr 172.21.15.151:0/1754354633 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db1ae1700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dac0ad160 0x7f2dac12ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 -- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dac0ad160 msgr2=0x7f2dac12ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dac0ad160 0x7f2dac12ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 -- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 msgr2=0x7f2dac1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 0x7f2dac1376f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 -- 172.21.15.151:0/1754354633 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0009420 con 0x7f2dac069f70 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db0adf700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 0x7f2dac1376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db12e0700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009ed0 tx=0x7f2da0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0011900 con 0x7f2dac069f70 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da0011a60 con 0x7f2dac069f70 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac1397b0 con 0x7f2dac069f70 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da001a680 con 0x7f2dac069f70 2023-09-30T02:03:01.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac12bf60 con 0x7f2dac069f70 2023-09-30T02:03:01.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.218+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac12c150 con 0x7f2dac069f70 2023-09-30T02:03:01.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.222+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2da001aa70 con 0x7f2dac069f70 2023-09-30T02:03:01.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.222+0000 7f2da97fa700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 0x7f2d98062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.222+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2da007e1d0 con 0x7f2dac069f70 2023-09-30T02:03:01.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.222+0000 7f2db0adf700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 0x7f2d98062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.222+0000 7f2db0adf700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 0x7f2d98062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dac139120 tx=0x7f2d9c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.230+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2da004e9b0 con 0x7f2dac069f70 2023-09-30T02:03:01.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dac138150 con 0x7f2d98060550 2023-09-30T02:03:01.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2da97fa700 1 -- 172.21.15.151:0/1754354633 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2d9c00f040 con 0x7f2d98060550 2023-09-30T02:03:01.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 msgr2=0x7f2d98062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 0x7f2d98062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2dac139120 tx=0x7f2d9c00b040 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 msgr2=0x7f2dac1384a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009ed0 tx=0x7f2da0017000 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 shutdown_connections 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f2d98060550 0x7f2d98062a00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2dac0ad160 0x7f2dac12ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2dac0acd90 0x7f2dac1376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 --2- 172.21.15.151:0/1754354633 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2dac069f70 0x7f2dac1384a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 >> 172.21.15.151:0/1754354633 conn(0x7f2dac0708b0 msgr2=0x7f2dac06ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 shutdown_connections 2023-09-30T02:03:01.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.390+0000 7f2db3544700 1 -- 172.21.15.151:0/1754354633 wait complete. 2023-09-30T02:03:01.395 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T02:03:01.404 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T02:03:01.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:03:01.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 Processor -- start 2023-09-30T02:03:01.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 -- start start 2023-09-30T02:03:01.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 0x7f940c13a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c0a5bb0 con 0x7f940c0a88a0 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c0a6c80 con 0x7f940c0aa970 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.510+0000 7f940bfff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0a56d0 con 0x7f940c139880 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56490/0 (socket says 172.21.15.151:56490) 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 -- 172.21.15.151:0/4093542631 learned_addr learned my addr 172.21.15.151:0/4093542631 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940a7fc700 1 --2- 172.21.15.151:0/4093542631 >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 0x7f940c13a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 -- 172.21.15.151:0/4093542631 >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 msgr2=0x7f940c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 --2- 172.21.15.151:0/4093542631 >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 0x7f940c13a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 -- 172.21.15.151:0/4093542631 >> v1:172.21.15.172:6789/0 conn(0x7f940c0a88a0 legacy=0x7f940c0a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 -- 172.21.15.151:0/4093542631 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0a6e80 con 0x7f940c139880 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940a7fc700 1 --2- 172.21.15.151:0/4093542631 >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 0x7f940c13a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940affd700 1 --2- 172.21.15.151:0/4093542631 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94000099a0 tx=0x7f9400009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=765c5304f52d5019 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f9409ffb700 1 -- 172.21.15.151:0/4093542631 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000b860 con 0x7f940c139880 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f9409ffb700 1 -- 172.21.15.151:0/4093542631 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940000b9c0 con 0x7f940c139880 2023-09-30T02:03:01.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f9409ffb700 1 -- 172.21.15.151:0/4093542631 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000bd80 con 0x7f940c139880 2023-09-30T02:03:01.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- 172.21.15.151:0/4093542631 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 msgr2=0x7f940c139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 --2- 172.21.15.151:0/4093542631 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94000099a0 tx=0x7f9400009250 comp rx=0 tx=0).stop 2023-09-30T02:03:01.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- 172.21.15.151:0/4093542631 shutdown_connections 2023-09-30T02:03:01.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 --2- 172.21.15.151:0/4093542631 >> v2:172.21.15.172:3300/0 conn(0x7f940c0aa970 0x7f940c13a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 --2- 172.21.15.151:0/4093542631 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c139880 0x7f940c139c50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- 172.21.15.151:0/4093542631 >> 172.21.15.151:0/4093542631 conn(0x7f940c070fe0 msgr2=0x7f940c0af770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- 172.21.15.151:0/4093542631 shutdown_connections 2023-09-30T02:03:01.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- 172.21.15.151:0/4093542631 wait complete. 2023-09-30T02:03:01.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 Processor -- start 2023-09-30T02:03:01.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.514+0000 7f940bfff700 1 -- start start 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 0x7f940c14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f940c0aa970 0x7f940c138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f940c139880 0x7f940c150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c13cc30 con 0x7f940c139880 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c14db80 con 0x7f940c0aa970 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0af0f0 con 0x7f940c0a88a0 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 0x7f940c14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 0x7f940c14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56498/0 (socket says 172.21.15.151:56498) 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940b7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f940c139880 0x7f940c150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 -- 172.21.15.151:0/2752599829 learned_addr learned my addr 172.21.15.151:0/2752599829 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940a7fc700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f940c0aa970 0x7f940c138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 -- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f940c0aa970 msgr2=0x7f940c138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f940c0aa970 0x7f940c138420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 -- 172.21.15.151:0/2752599829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f940c139880 msgr2=0x7f940c150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f940c139880 0x7f940c150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9400009490 con 0x7f940c0a88a0 2023-09-30T02:03:01.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940affd700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 0x7f940c14d540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94000090b0 tx=0x7f9400017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940001f070 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9400002f30 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000bd80 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c0af310 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.518+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c150c60 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f940c0670a0 con 0x7f940c0a88a0 2023-09-30T02:03:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9400023070 con 0x7f940c0a88a0 2023-09-30T02:03:01.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f93f80605d0 0x7f93f8062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f940007d5e0 con 0x7f940c0a88a0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f940a7fc700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f93f80605d0 0x7f93f8062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7f93f80669a0 0x7f93f8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93f8069460 con 0x7f93f80669a0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f940007fac0 con 0x7f940c0a88a0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f940b7fe700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7f93f80669a0 0x7f93f8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f940a7fc700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f93f80605d0 0x7f93f8062a80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f940c13c880 tx=0x7f93fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.522+0000 7f940b7fe700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7f93f80669a0 0x7f93f8068db0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.526+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== osd.7 v2:172.21.15.172:6808/31450 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f93f8069460 con 0x7f93f80669a0 2023-09-30T02:03:01.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 --> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f940c0a4fb0 con 0x7f93f80669a0 2023-09-30T02:03:01.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f93f2ffd700 1 -- 172.21.15.151:0/2752599829 <== osd.7 v2:172.21.15.172:6808/31450 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f940c0a4fb0 con 0x7f93f80669a0 2023-09-30T02:03:01.545 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2023-09-30T02:03:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:03:01.545 INFO:teuthology.orchestra.run.smithi151.stdout:} 2023-09-30T02:03:01.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7f93f80669a0 msgr2=0x7f93f8068db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6808/31450,v1:172.21.15.172:6809/31450] conn(0x7f93f80669a0 0x7f93f8068db0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f93f80605d0 msgr2=0x7f93f8062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f93f80605d0 0x7f93f8062a80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f940c13c880 tx=0x7f93fc008040 comp rx=0 tx=0).stop 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 msgr2=0x7f940c14d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 --2- 172.21.15.151:0/2752599829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f940c0a88a0 0x7f940c14d540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94000090b0 tx=0x7f9400017000 comp rx=0 tx=0).stop 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940b7fe700 1 -- 172.21.15.151:0/2752599829 reap_dead start 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 shutdown_connections 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 >> 172.21.15.151:0/2752599829 conn(0x7f940c070fe0 msgr2=0x7f940c0a7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 shutdown_connections 2023-09-30T02:03:01.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.542+0000 7f940bfff700 1 -- 172.21.15.151:0/2752599829 wait complete. 2023-09-30T02:03:01.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T02:03:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 Processor -- start 2023-09-30T02:03:01.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 -- start start 2023-09-30T02:03:01.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 0x7fdae00ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdae013e4e0 0x7fdae013e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00abe00 con 0x7fdae006b550 2023-09-30T02:03:01.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae006c700 con 0x7fdae013e4e0 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadf59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00ac820 con 0x7fdae006b180 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdaddd9b700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fdae006b550 0x7fdae01415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48794/0 (socket says 172.21.15.151:48794) 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 0x7fdae00ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdaddd9b700 1 -- 172.21.15.151:0/3308538929 learned_addr learned my addr 172.21.15.151:0/3308538929 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdaded9d700 1 --2- 172.21.15.151:0/3308538929 >> v2:172.21.15.172:3300/0 conn(0x7fdae013e4e0 0x7fdae013e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadd59a700 1 -- 172.21.15.151:0/3308538929 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104755278 0 0) 0x7fdae00abe00 con 0x7fdae006b550 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadd59a700 1 -- 172.21.15.151:0/3308538929 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc003570 con 0x7fdae006b550 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 -- 172.21.15.151:0/3308538929 >> v2:172.21.15.172:3300/0 conn(0x7fdae013e4e0 msgr2=0x7fdae013e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 --2- 172.21.15.151:0/3308538929 >> v2:172.21.15.172:3300/0 conn(0x7fdae013e4e0 0x7fdae013e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 -- 172.21.15.151:0/3308538929 >> v1:172.21.15.151:6789/0 conn(0x7fdae006b550 legacy=0x7fdae01415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 -- 172.21.15.151:0/3308538929 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae006c8e0 con 0x7fdae006b180 2023-09-30T02:03:01.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdade59c700 1 --2- 172.21.15.151:0/3308538929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 0x7fdae00ac2e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdad00098e0 tx=0x7fdad00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9c3a2d2cf0e749 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadd59a700 1 -- 172.21.15.151:0/3308538929 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0011630 con 0x7fdae006b180 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadd59a700 1 -- 172.21.15.151:0/3308538929 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad0011790 con 0x7fdae006b180 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.674+0000 7fdadd59a700 1 -- 172.21.15.151:0/3308538929 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0011b30 con 0x7fdae006b180 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- 172.21.15.151:0/3308538929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 msgr2=0x7fdae00ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- 172.21.15.151:0/3308538929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 0x7fdae00ac2e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdad00098e0 tx=0x7fdad00091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- 172.21.15.151:0/3308538929 shutdown_connections 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- 172.21.15.151:0/3308538929 >> v2:172.21.15.172:3300/0 conn(0x7fdae013e4e0 0x7fdae013e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- 172.21.15.151:0/3308538929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae006b180 0x7fdae00ac2e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- 172.21.15.151:0/3308538929 >> 172.21.15.151:0/3308538929 conn(0x7fdae0070510 msgr2=0x7fdae0070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- 172.21.15.151:0/3308538929 shutdown_connections 2023-09-30T02:03:01.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- 172.21.15.151:0/3308538929 wait complete. 2023-09-30T02:03:01.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 Processor -- start 2023-09-30T02:03:01.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- start start 2023-09-30T02:03:01.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae006b180 0x7fdae01436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 0x7fdae0143cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 0x7fdae013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae0144450 con 0x7fdae006b550 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae01428f0 con 0x7fdae006b180 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdadf59e700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae01448e0 con 0x7fdae013e4e0 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 0x7fdae0143cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 0x7fdae013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaddd9b700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 0x7fdae0143cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:45060/0 (socket says 172.21.15.151:45060) 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaddd9b700 1 -- 172.21.15.151:0/2427078766 learned_addr learned my addr 172.21.15.151:0/2427078766 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdade59c700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae006b180 0x7fdae01436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 -- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae006b180 msgr2=0x7fdae01436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae006b180 0x7fdae01436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 -- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 msgr2=0x7fdae0143cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 0x7fdae0143cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 -- 172.21.15.151:0/2427078766 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad0009420 con 0x7fdae013e4e0 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdaded9d700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 0x7fdae013f870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdad800e900 tx=0x7fdad80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8013680 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.678+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad8013820 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8013be0 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae00a8f10 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae013ffd0 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae0145920 con 0x7fdae013e4e0 2023-09-30T02:03:01.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdad80123d0 con 0x7fdae013e4e0 2023-09-30T02:03:01.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.682+0000 7fdad67fc700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 0x7fdac0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.686+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fdad807c3d0 con 0x7fdae013e4e0 2023-09-30T02:03:01.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.686+0000 7fdade59c700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 0x7fdac0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.686+0000 7fdade59c700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 0x7fdac0062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdad0011fd0 tx=0x7fdad0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.694+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdad80505a0 con 0x7fdae013e4e0 2023-09-30T02:03:01.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.854+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fdae01445c0 con 0x7fdac0060550 2023-09-30T02:03:01.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.854+0000 7fdad67fc700 1 -- 172.21.15.151:0/2427078766 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdae01445c0 con 0x7fdac0060550 2023-09-30T02:03:01.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 msgr2=0x7fdac0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 0x7fdac0062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdad0011fd0 tx=0x7fdad0017000 comp rx=0 tx=0).stop 2023-09-30T02:03:01.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 msgr2=0x7fdae013f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 0x7fdae013f870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdad800e900 tx=0x7fdad80091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 shutdown_connections 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fdac0060550 0x7fdac0062a00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdae013e4e0 0x7fdae013f870 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdae006b550 0x7fdae0143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 --2- 172.21.15.151:0/2427078766 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae006b180 0x7fdae01436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 >> 172.21.15.151:0/2427078766 conn(0x7fdae0070510 msgr2=0x7fdae00ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 shutdown_connections 2023-09-30T02:03:01.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.858+0000 7fdadf59e700 1 -- 172.21.15.151:0/2427078766 wait complete. 2023-09-30T02:03:01.862 INFO:teuthology.orchestra.run.smithi151.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T02:03:01.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:03:01.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 Processor -- start 2023-09-30T02:03:01.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- start start 2023-09-30T02:03:01.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 0x7f477c0aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 0x7f477c0ae770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c0a9c40 con 0x7f477c0b0910 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c0b0ce0 con 0x7f477c0a8850 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0a9eb0 con 0x7f477c0a8c90 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56532/0 (socket says 172.21.15.151:56532) 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 -- 172.21.15.151:0/1328190083 learned_addr learned my addr 172.21.15.151:0/1328190083 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781719700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 0x7f477c0aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4780f18700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 0x7f477c0ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 -- 172.21.15.151:0/1328190083 >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 msgr2=0x7f477c0ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 0x7f477c0ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 -- 172.21.15.151:0/1328190083 >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 msgr2=0x7f477c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 0x7f477c0aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 -- 172.21.15.151:0/1328190083 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c0aa190 con 0x7f477c0a8c90 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4780f18700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 0x7f477c0ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781719700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 0x7f477c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4781f1a700 1 --2- 172.21.15.151:0/1328190083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47780099b0 tx=0x7f4778009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d69871860a4f32e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4773fff700 1 -- 172.21.15.151:0/1328190083 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800b820 con 0x7f477c0a8c90 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4773fff700 1 -- 172.21.15.151:0/1328190083 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477800b980 con 0x7f477c0a8c90 2023-09-30T02:03:01.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f4773fff700 1 -- 172.21.15.151:0/1328190083 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800bd20 con 0x7f477c0a8c90 2023-09-30T02:03:01.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- 172.21.15.151:0/1328190083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 msgr2=0x7f477c0aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- 172.21.15.151:0/1328190083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f47780099b0 tx=0x7f4778009250 comp rx=0 tx=0).stop 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- 172.21.15.151:0/1328190083 shutdown_connections 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- 172.21.15.151:0/1328190083 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c0aecb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.172:3300/0 conn(0x7f477c0a8850 0x7f477c0ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 --2- 172.21.15.151:0/1328190083 >> v2:172.21.15.151:3300/0 conn(0x7f477c0b0910 0x7f477c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- 172.21.15.151:0/1328190083 >> 172.21.15.151:0/1328190083 conn(0x7f477c070510 msgr2=0x7f477c070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- 172.21.15.151:0/1328190083 shutdown_connections 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.978+0000 7f478397d700 1 -- 172.21.15.151:0/1328190083 wait complete. 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 Processor -- start 2023-09-30T02:03:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 -- start start 2023-09-30T02:03:01.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 0x7f477c141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f477c0b0910 0x7f477c150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c11c740 con 0x7f477c0b0910 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c14db80 con 0x7f477c0a8850 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f478397d700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c11c940 con 0x7f477c0a8c90 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f4780f18700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f4781f1a700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f477c0b0910 0x7f477c150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f4780f18700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:56544/0 (socket says 172.21.15.151:56544) 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f4780f18700 1 -- 172.21.15.151:0/49367247 learned_addr learned my addr 172.21.15.151:0/49367247 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.982+0000 7f4781719700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 0x7f477c141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 -- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 msgr2=0x7f477c141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 0x7f477c141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 -- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f477c0b0910 msgr2=0x7f477c150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f477c0b0910 0x7f477c150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 -- 172.21.15.151:0/49367247 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780094f0 con 0x7f477c0a8c90 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4781719700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 0x7f477c141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f4780f18700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f477400cfb0 tx=0x7f4774007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477400ae30 con 0x7f477c0a8c90 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774016420 con 0x7f477c0a8c90 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c14d720 con 0x7f477c0a8c90 2023-09-30T02:03:01.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740167e0 con 0x7f477c0a8c90 2023-09-30T02:03:01.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c150c60 con 0x7f477c0a8c90 2023-09-30T02:03:01.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c150220 con 0x7f477c0a8c90 2023-09-30T02:03:01.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.986+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4774016bd0 con 0x7f477c0a8c90 2023-09-30T02:03:01.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.990+0000 7f47717fa700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 0x7f4760062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:01.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.990+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4774012070 con 0x7f477c0a8c90 2023-09-30T02:03:01.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.990+0000 7f4781719700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 0x7f4760062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:01.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.990+0000 7f4781719700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 0x7f4760062a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f476c006f00 tx=0x7f476c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:01.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:01.998+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f477404cbe0 con 0x7f477c0a8c90 2023-09-30T02:03:02.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.150+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f477c0670a0 con 0x7f4760060550 2023-09-30T02:03:02.157 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.154+0000 7f47717fa700 1 -- 172.21.15.151:0/49367247 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70445 (secure 0 0 0) 0x7f477c0670a0 con 0x7f4760060550 2023-09-30T02:03:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:03:02.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 msgr2=0x7f4760062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:02.160 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 0x7f4760062a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f476c006f00 tx=0x7f476c008040 comp rx=0 tx=0).stop 2023-09-30T02:03:02.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 msgr2=0x7f477c14d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:02.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f477400cfb0 tx=0x7f4774007870 comp rx=0 tx=0).stop 2023-09-30T02:03:02.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 shutdown_connections 2023-09-30T02:03:02.161 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f4760060550 0x7f4760062a00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f477c0b0910 0x7f477c150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f477c0a8c90 0x7f477c14d1e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 --2- 172.21.15.151:0/49367247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f477c0a8850 0x7f477c141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 >> 172.21.15.151:0/49367247 conn(0x7f477c070510 msgr2=0x7f477c13f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 shutdown_connections 2023-09-30T02:03:02.163 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:02.158+0000 7f478397d700 1 -- 172.21.15.151:0/49367247 wait complete. 2023-09-30T02:03:02.163 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T02:03:02.172 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2023-09-30T02:03:01.529774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":43136,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":40640,"kb_avail":754931584,"statfs":{"total":773094113280,"available":773049942016,"internally_reserved":0,"allocated":2490368,"data_stored":1519970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41615360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003422"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553615+0000","last_change":"2023-09-30T01:57:50.548644+0000","last_active":"2023-09-30T01:57:50.553615+0000","last_peered":"2023-09-30T01:57:50.553615+0000","last_clean":"2023-09-30T01:57:50.553615+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T01:57:50.553615+0000","last_undegraded":"2023-09-30T01:57:50.553615+0000","last_fullsized":"2023-09-30T01:57:50.553615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:56:39.564553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.554354+0000","last_change":"2023-09-30T01:57:50.543328+0000","last_active":"2023-09-30T01:57:51.554354+0000","last_peered":"2023-09-30T01:57:51.554354+0000","last_clean":"2023-09-30T01:57:51.554354+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T01:57:51.554354+0000","last_undegraded":"2023-09-30T01:57:51.554354+0000","last_fullsized":"2023-09-30T01:57:51.554354+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.111495+0000","last_change":"2023-09-30T01:57:51.109930+0000","last_active":"2023-09-30T01:57:51.111495+0000","last_peered":"2023-09-30T01:57:51.111495+0000","last_clean":"2023-09-30T01:57:51.111495+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T01:57:51.111495+0000","last_undegraded":"2023-09-30T01:57:51.111495+0000","last_fullsized":"2023-09-30T01:57:51.111495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:46:38.135479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:00.484006+0000","last_change":"2023-09-30T02:03:00.483913+0000","last_active":"2023-09-30T02:03:00.484006+0000","last_peered":"2023-09-30T02:03:00.484006+0000","last_clean":"2023-09-30T02:03:00.484006+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T02:03:00.484006+0000","last_undegraded":"2023-09-30T02:03:00.484006+0000","last_fullsized":"2023-09-30T02:03:00.484006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:00.483582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:00.483582+0000","last_clean_scrub_stamp":"2023-09-30T02:03:00.483582+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:22:38.027340+0000","scrub_duration":0.0039998719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.551200+0000","last_change":"2023-09-30T01:57:50.547452+0000","last_active":"2023-09-30T01:57:50.551200+0000","last_peered":"2023-09-30T01:57:50.551200+0000","last_clean":"2023-09-30T01:57:50.551200+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T01:57:50.551200+0000","last_undegraded":"2023-09-30T01:57:50.551200+0000","last_fullsized":"2023-09-30T01:57:50.551200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.857828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.702970+0000","last_change":"2023-09-30T01:57:50.701554+0000","last_active":"2023-09-30T01:57:50.702970+0000","last_peered":"2023-09-30T01:57:50.702970+0000","last_clean":"2023-09-30T01:57:50.702970+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T01:57:50.702970+0000","last_undegraded":"2023-09-30T01:57:50.702970+0000","last_fullsized":"2023-09-30T01:57:50.702970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:28.055327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.549262+0000","last_change":"2023-09-30T01:57:50.547375+0000","last_active":"2023-09-30T01:57:50.549262+0000","last_peered":"2023-09-30T01:57:50.549262+0000","last_clean":"2023-09-30T01:57:50.549262+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T01:57:50.549262+0000","last_undegraded":"2023-09-30T01:57:50.549262+0000","last_fullsized":"2023-09-30T01:57:50.549262+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:09.262916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:50.553788+0000","last_change":"2023-09-30T01:57:50.549603+0000","last_active":"2023-09-30T01:57:50.553788+0000","last_peered":"2023-09-30T01:57:50.553788+0000","last_clean":"2023-09-30T01:57:50.553788+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T01:57:50.553788+0000","last_undegraded":"2023-09-30T01:57:50.553788+0000","last_fullsized":"2023-09-30T01:57:50.553788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:49.517644+0000","last_clean_scrub_stamp":"2023-09-30T01:57:49.517644+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:02:06.151614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:57:51.129346+0000","last_change":"2023-09-30T01:57:49.480715+0000","last_active":"2023-09-30T01:57:51.129346+0000","last_peered":"2023-09-30T01:57:51.129346+0000","last_clean":"2023-09-30T01:57:51.129346+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T01:57:51.129346+0000","last_undegraded":"2023-09-30T01:57:51.129346+0000","last_fullsized":"2023-09-30T01:57:51.129346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:57:47.500496+0000","last_clean_scrub_stamp":"2023-09-30T01:57:47.500496+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:52:46.008154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.86799999999999999,"15min":0.872},"min":{"1min":0.57199999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.93899999999999995,"15min":0.997},"min":{"1min":0.64800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.149,"5min":1.353,"15min":1.353},"last":1.0640000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.91900000000000004,"15min":0.93700000000000006},"min":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3620000000000001,"5min":1.365,"15min":1.365},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91000000000000003,"15min":0.89500000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97699999999999998,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.619,"15min":1.619},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.94499999999999995,"15min":0.97299999999999998},"min":{"1min":0.69099999999999995,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.73299999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98799999999999999,"15min":1.012},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3,"5min":1.446,"15min":1.446},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.103,"15min":1.089},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.032}]},{"osd":4,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.91400000000000003,"15min":0.91800000000000004},"min":{"1min":0.63,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1779999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.054,"5min":1.083,"15min":1.091},"min":{"1min":0.76500000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3879999999999999,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.91700000000000004}]},{"osd":5,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.03,"15min":1.048},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2709999999999999,"5min":1.55,"15min":1.55},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.044,"5min":1.077,"15min":1.093},"min":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.377,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.78100000000000003}]},{"osd":7,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.157,"15min":1.1899999999999999},"min":{"1min":0.93200000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.3759999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.91400000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.601,"5min":1.627,"15min":1.627},"last":0.95899999999999996}]}]},{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.91200000000000003,"15min":0.89800000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.264,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.877,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.1830000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.70299999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1319999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.054,"5min":1.0369999999999999,"15min":1.0609999999999999},"min":{"1min":0.86499999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.28,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.92400000000000004}]},{"osd":2,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.67300000000000004,"5min":0.63,"15min":0.63},"max":{"1min":1.0489999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.006,"5min":0.94599999999999995,"15min":0.94899999999999995},"min":{"1min":0.72899999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.27,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.73899999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94999999999999996,"15min":0.96599999999999997},"min":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2,"5min":1.218,"15min":1.218},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0029999999999999,"15min":1.018},"min":{"1min":0.85199999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.208,"5min":1.365,"15min":1.365},"last":0.878}]},{"osd":4,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.90300000000000002,"15min":0.94099999999999995},"min":{"1min":0.71899999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.069},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.0980000000000001},"min":{"1min":0.76300000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.258,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.111}]},{"osd":5,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.04,"15min":1.069},"min":{"1min":0.79700000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4419999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.129,"15min":1.1539999999999999},"min":{"1min":1.002,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.323,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":0.98599999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.123,"15min":1.1160000000000001},"min":{"1min":0.89500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.343,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.171,"15min":1.1859999999999999},"min":{"1min":1.0169999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.1919999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83799999999999997,"15min":0.83599999999999997},"min":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.98099999999999998,"15min":1.026},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.751}]},{"osd":1,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85999999999999999,"15min":0.85599999999999998},"min":{"1min":0.66100000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.262,"15min":1.262},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.64400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.92800000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.97899999999999998,"15min":1.028},"min":{"1min":0.66300000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.214,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.242,"5min":1.575,"15min":1.575},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.95099999999999996},"min":{"1min":0.66900000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.79500000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90700000000000003,"15min":0.89400000000000002},"min":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.129},{"interface":"front","average":{"1min":1.028,"5min":1.04,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.216,"5min":1.329,"15min":1.329},"last":1.0720000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1950000000000001,"5min":1.669,"15min":1.669},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.103,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.2949999999999999,"5min":1.75,"15min":1.75},"last":1.0409999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:01:56 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1970000000000001,"15min":1.23},"min":{"1min":1.03,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.349,"5min":1.859,"15min":1.859},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.034,"15min":1.032},"min":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3799999999999999,"5min":1.526,"15min":1.526},"last":1.161}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":0.94099999999999995},"min":{"1min":0.67500000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1970000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.97599999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1890000000000001,"5min":1.327,"15min":1.327},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.131,"15min":1.167},"min":{"1min":0.96499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.4410000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.268}]},{"osd":2,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.008},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.95699999999999996,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.163,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.91500000000000004}]},{"osd":3,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.78600000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.3500000000000001,"5min":1.472,"15min":1.472},"last":1.165},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.997,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.4510000000000001,"5min":3.9889999999999999,"15min":3.9889999999999999},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.89900000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.996,"5min":1.292,"15min":1.292},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.113},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4099999999999999,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.1040000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1020000000000001,"15min":1.048},"min":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.85999999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.2549999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.0700000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1319999999999999,"15min":1.129},"min":{"1min":0.96699999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.5549999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.873},{"interface":"front","average":{"1min":1.099,"5min":1.1519999999999999,"15min":1.1679999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3100000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.133}]}]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91200000000000003,"5min":2.29,"15min":2.29},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78300000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1659999999999999,"5min":1.569,"15min":1.569},"last":0.73299999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":1.28,"15min":1.28},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.84899999999999998,"15min":0.88800000000000001},"min":{"1min":0.621,"5min":0.499,"15min":0.499},"max":{"1min":1.3600000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.88600000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.0880000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1200000000000001,"5min":1.363,"15min":1.363},"last":0.82899999999999996}]},{"osd":4,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0700000000000001,"15min":1.1459999999999999},"min":{"1min":0.79400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.351,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.103},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0720000000000001,"15min":1.0840000000000001},"min":{"1min":0.85799999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.532,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.079}]},{"osd":5,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.046,"15min":1.0800000000000001},"min":{"1min":0.83199999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.474,"5min":1.645,"15min":1.645},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.80800000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.371,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.2050000000000001,"15min":1.2749999999999999},"min":{"1min":0.79700000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.704,"5min":2.516,"15min":2.516},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.206,"15min":1.2509999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6679999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.1659999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":1.002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.524,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3140000000000001},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.4370000000000001,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.141}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5168,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366672,"statfs":{"total":96636764160,"available":96631472128,"internally_reserved":0,"allocated":172032,"data_stored":56052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65600000000000003,"15min":0.65300000000000002},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.034,"5min":1.181,"15min":1.181},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72699999999999998,"15min":0.751},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.1459999999999999,"5min":1.284,"15min":1.284},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":1.135,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84199999999999997,"15min":0.85899999999999999},"min":{"1min":0.59099999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.98199999999999998,"15min":1.03},"min":{"1min":0.72799999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.113,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.024,"15min":1.0780000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.359,"5min":1.819,"15min":1.819},"last":1.3089999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0569999999999999,"15min":1.085},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5229999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0720000000000001,"15min":1.101},"min":{"1min":0.77000000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.4379999999999999,"5min":1.659,"15min":1.659},"last":1.3440000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:01:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.163},"min":{"1min":0.79600000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.667,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.149,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.3839999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.4259999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.149,"15min":1.155},"min":{"1min":1.032,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4179999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.2130000000000001,"15min":1.226},"min":{"1min":0.85299999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.4690000000000001,"5min":1.79,"15min":1.79},"last":1.2749999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91300000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.76600000000000001},"min":{"1min":0.56899999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.113,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.0189999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68300000000000005,"15min":0.72999999999999998},"min":{"1min":0.52600000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.90500000000000003,"5min":1.296,"15min":1.296},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.81299999999999994,"15min":0.86599999999999999},"min":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.252,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.71999999999999997}]},{"osd":3,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.84099999999999997,"15min":0.874},"min":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0660000000000001,"5min":1.218,"15min":1.218},"last":0.88600000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.97799999999999998,"15min":0.95999999999999996},"min":{"1min":0.79200000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.343,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.083,"5min":1.0820000000000001,"15min":1.1160000000000001},"min":{"1min":0.88800000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.734,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.1830000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0389999999999999,"15min":1.1200000000000001},"min":{"1min":0.76200000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.2410000000000001,"5min":1.625,"15min":1.625},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.119,"15min":1.165},"min":{"1min":0.82399999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.6639999999999999,"5min":1.893,"15min":1.893},"last":1.234}]},{"osd":6,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.133,"15min":1.1930000000000001},"min":{"1min":0.84199999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.2989999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1299999999999999,"15min":1.181},"min":{"1min":0.86099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.369,"5min":1.778,"15min":1.778},"last":0.98799999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1850000000000001,"15min":1.232},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.6240000000000001,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.1899999999999999,"15min":1.2529999999999999},"min":{"1min":0.97199999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.3300000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.107}]}]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":585928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66700000000000004,"15min":0.64800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.871,"5min":1.006,"15min":1.006},"last":0.749},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77200000000000002,"15min":0.755},"min":{"1min":0.45800000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.083,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.95599999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88,"5min":1.087,"15min":1.087},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.81699999999999995,"15min":0.79000000000000004},"min":{"1min":0.54300000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.9729999999999999,"5min":3.9729999999999999,"15min":3.9729999999999999},"last":0.77500000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72299999999999998,"15min":0.69899999999999995},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.85199999999999998,"15min":0.874},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.99399999999999999,"15min":0.98599999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.194,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.0189999999999999,"15min":1.0429999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.288,"5min":1.448,"15min":1.448},"last":1.0229999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.98099999999999998,"15min":1.0449999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.244,"5min":1.468,"15min":1.468},"last":0.997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0629999999999999,"15min":1.109},"min":{"1min":0.79100000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2050000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0620000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:02:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0920000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1899999999999999,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.147},"min":{"1min":0.81599999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.5609999999999999,"5min":1.768,"15min":1.768},"last":1.226}]},{"osd":7,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.147,"15min":1.165},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.415,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.171,"15min":1.208},"min":{"1min":0.877,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5329999999999999,"5min":1.669,"15min":1.669},"last":1.2969999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:03:02.177 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.178 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.178 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T01:57:49.517644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=49, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T01:57:47.500496+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=57, tm_sec=47, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=2, tm_sec=57, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:03:02.179 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T02:03:22.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:03:22.295 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 Processor -- start 2023-09-30T02:03:22.295 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- start start 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 0x7fe138137b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 0x7fe138138530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1380ab550 con 0x7fe138070740 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1380ac130 con 0x7fe13813edc0 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe138138a70 con 0x7fe1381380b0 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 0x7fe138138530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13d54f700 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 0x7fe138137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13dd50700 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe138070740 0x7fe13813e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.151:57250/0 (socket says 172.21.15.151:57250) 2023-09-30T02:03:22.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13dd50700 1 -- 172.21.15.151:0/3214097371 learned_addr learned my addr 172.21.15.151:0/3214097371 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:03:22.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 -- 172.21.15.151:0/3214097371 >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 msgr2=0x7fe138137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 --2- 172.21.15.151:0/3214097371 >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 0x7fe138137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 -- 172.21.15.151:0/3214097371 >> v1:172.21.15.172:6789/0 conn(0x7fe138070740 legacy=0x7fe13813e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 -- 172.21.15.151:0/3214097371 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe138070b10 con 0x7fe1381380b0 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13cd4e700 1 -- 172.21.15.151:0/3214097371 <== mon.1 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101642724 0 0) 0x7fe1380ab550 con 0x7fe138070740 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13d54f700 1 --2- 172.21.15.151:0/3214097371 >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 0x7fe138137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13e551700 1 --2- 172.21.15.151:0/3214097371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 0x7fe138138530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe130009e90 tx=0x7fe130009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7268f36246e37f93 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13cd4e700 1 -- 172.21.15.151:0/3214097371 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13000e070 con 0x7fe1381380b0 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13cd4e700 1 -- 172.21.15.151:0/3214097371 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13000bbf0 con 0x7fe1381380b0 2023-09-30T02:03:22.298 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13cd4e700 1 -- 172.21.15.151:0/3214097371 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe130012430 con 0x7fe1381380b0 2023-09-30T02:03:22.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- 172.21.15.151:0/3214097371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 msgr2=0x7fe138138530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 --2- 172.21.15.151:0/3214097371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 0x7fe138138530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe130009e90 tx=0x7fe130009780 comp rx=0 tx=0).stop 2023-09-30T02:03:22.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 -- 172.21.15.151:0/3214097371 shutdown_connections 2023-09-30T02:03:22.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 --2- 172.21.15.151:0/3214097371 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1381380b0 0x7fe138138530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.294+0000 7fe13ed52700 1 --2- 172.21.15.151:0/3214097371 >> v2:172.21.15.151:3300/0 conn(0x7fe13813edc0 0x7fe138137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.298+0000 7fe13ed52700 1 -- 172.21.15.151:0/3214097371 >> 172.21.15.151:0/3214097371 conn(0x7fe1380b03f0 msgr2=0x7fe1380b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.298+0000 7fe13ed52700 1 -- 172.21.15.151:0/3214097371 shutdown_connections 2023-09-30T02:03:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.298+0000 7fe13ed52700 1 -- 172.21.15.151:0/3214097371 wait complete. 2023-09-30T02:03:22.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.298+0000 7fe13ed52700 1 Processor -- start 2023-09-30T02:03:22.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.298+0000 7fe13ed52700 1 -- start start 2023-09-30T02:03:22.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.303 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1381380b0 0x7fe138138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 0x7fe138139380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380a6680 con 0x7fe13813edc0 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380a47c0 con 0x7fe1381380b0 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380a49d0 con 0x7fe138070740 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:38002/0 (socket says 172.21.15.151:38002) 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 -- 172.21.15.151:0/2200335427 learned_addr learned my addr 172.21.15.151:0/2200335427 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13e551700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 0x7fe138139380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13d54f700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1381380b0 0x7fe138138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 -- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1381380b0 msgr2=0x7fe138138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1381380b0 0x7fe138138e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 -- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 msgr2=0x7fe138139380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 0x7fe138139380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 -- 172.21.15.151:0/2200335427 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128009420 con 0x7fe138070740 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13e551700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 0x7fe138139380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:03:22.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13dd50700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c5e0 tx=0x7fe134001070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134005b30 con 0x7fe138070740 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe134005c90 con 0x7fe138070740 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe130009a20 con 0x7fe138070740 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134007300 con 0x7fe138070740 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1380a4ef0 con 0x7fe138070740 2023-09-30T02:03:22.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.302+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1380a5110 con 0x7fe138070740 2023-09-30T02:03:22.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.306+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe13401b3b0 con 0x7fe138070740 2023-09-30T02:03:22.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.306+0000 7fe12dffb700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 0x7fe11c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.306+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe13407ba90 con 0x7fe138070740 2023-09-30T02:03:22.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.306+0000 7fe13d54f700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 0x7fe11c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.306+0000 7fe13d54f700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 0x7fe11c062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe13813b7a0 tx=0x7fe1280091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.314+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe13404c1d0 con 0x7fe138070740 2023-09-30T02:03:22.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.470+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 --> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1380008d0 con 0x7fe11c060550 2023-09-30T02:03:22.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.478+0000 7fe12dffb700 1 -- 172.21.15.151:0/2200335427 <== mgr.4102 v2:172.21.15.172:6832/31144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70550 (secure 0 0 0) 0x7fe1380008d0 con 0x7fe11c060550 2023-09-30T02:03:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-09-30T02:03:22.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 msgr2=0x7fe11c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 0x7fe11c062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe13813b7a0 tx=0x7fe1280091a0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 msgr2=0x7fe13813b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c5e0 tx=0x7fe134001070 comp rx=0 tx=0).stop 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 shutdown_connections 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7fe11c060550 0x7fe11c062a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe13813edc0 0x7fe138139380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1381380b0 0x7fe138138e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 --2- 172.21.15.151:0/2200335427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe138070740 0x7fe13813b9b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 >> 172.21.15.151:0/2200335427 conn(0x7fe1380b03f0 msgr2=0x7fe1380aa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 shutdown_connections 2023-09-30T02:03:22.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.482+0000 7fe13ed52700 1 -- 172.21.15.151:0/2200335427 wait complete. 2023-09-30T02:03:22.485 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-09-30T02:03:22.496 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2023-09-30T02:03:21.535721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":43200,"kb_used_data":2432,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":754931520,"statfs":{"total":773094113280,"available":773049876480,"internally_reserved":0,"allocated":2490368,"data_stored":1519970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:02.119100+0000","last_change":"2023-09-30T02:03:02.119021+0000","last_active":"2023-09-30T02:03:02.119100+0000","last_peered":"2023-09-30T02:03:02.119100+0000","last_clean":"2023-09-30T02:03:02.119100+0000","last_became_active":"2023-09-30T01:57:50.548157+0000","last_became_peered":"2023-09-30T01:57:50.548157+0000","last_unstale":"2023-09-30T02:03:02.119100+0000","last_undegraded":"2023-09-30T02:03:02.119100+0000","last_fullsized":"2023-09-30T02:03:02.119100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:02.118745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:02.118745+0000","last_clean_scrub_stamp":"2023-09-30T02:03:02.118745+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:16.718355+0000","scrub_duration":0.0040702259999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:02.471055+0000","last_change":"2023-09-30T02:03:02.470924+0000","last_active":"2023-09-30T02:03:02.471055+0000","last_peered":"2023-09-30T02:03:02.471055+0000","last_clean":"2023-09-30T02:03:02.471055+0000","last_became_active":"2023-09-30T01:57:50.542965+0000","last_became_peered":"2023-09-30T01:57:50.542965+0000","last_unstale":"2023-09-30T02:03:02.471055+0000","last_undegraded":"2023-09-30T02:03:02.471055+0000","last_fullsized":"2023-09-30T02:03:02.471055+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:02.470648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:02.470648+0000","last_clean_scrub_stamp":"2023-09-30T02:03:02.470648+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:09.506021+0000","scrub_duration":0.0042299440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:07.492912+0000","last_change":"2023-09-30T02:03:07.492836+0000","last_active":"2023-09-30T02:03:07.492912+0000","last_peered":"2023-09-30T02:03:07.492912+0000","last_clean":"2023-09-30T02:03:07.492912+0000","last_became_active":"2023-09-30T01:57:51.109233+0000","last_became_peered":"2023-09-30T01:57:51.109233+0000","last_unstale":"2023-09-30T02:03:07.492912+0000","last_undegraded":"2023-09-30T02:03:07.492912+0000","last_fullsized":"2023-09-30T02:03:07.492912+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:07.492584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:07.492584+0000","last_clean_scrub_stamp":"2023-09-30T02:03:07.492584+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:24:59.393150+0000","scrub_duration":0.0039585180000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:00.484006+0000","last_change":"2023-09-30T02:03:00.483913+0000","last_active":"2023-09-30T02:03:00.484006+0000","last_peered":"2023-09-30T02:03:00.484006+0000","last_clean":"2023-09-30T02:03:00.484006+0000","last_became_active":"2023-09-30T01:57:51.106414+0000","last_became_peered":"2023-09-30T01:57:51.106414+0000","last_unstale":"2023-09-30T02:03:00.484006+0000","last_undegraded":"2023-09-30T02:03:00.484006+0000","last_fullsized":"2023-09-30T02:03:00.484006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:00.483582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:00.483582+0000","last_clean_scrub_stamp":"2023-09-30T02:03:00.483582+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:22:38.027340+0000","scrub_duration":0.0039998719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:01.674183+0000","last_change":"2023-09-30T02:03:01.674108+0000","last_active":"2023-09-30T02:03:01.674183+0000","last_peered":"2023-09-30T02:03:01.674183+0000","last_clean":"2023-09-30T02:03:01.674183+0000","last_became_active":"2023-09-30T01:57:50.547110+0000","last_became_peered":"2023-09-30T01:57:50.547110+0000","last_unstale":"2023-09-30T02:03:01.674183+0000","last_undegraded":"2023-09-30T02:03:01.674183+0000","last_fullsized":"2023-09-30T02:03:01.674183+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:01.673838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:01.673838+0000","last_clean_scrub_stamp":"2023-09-30T02:03:01.673838+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:28.507790+0000","scrub_duration":0.003884036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:01.091549+0000","last_change":"2023-09-30T02:03:01.091511+0000","last_active":"2023-09-30T02:03:01.091549+0000","last_peered":"2023-09-30T02:03:01.091549+0000","last_clean":"2023-09-30T02:03:01.091549+0000","last_became_active":"2023-09-30T01:57:50.700928+0000","last_became_peered":"2023-09-30T01:57:50.700928+0000","last_unstale":"2023-09-30T02:03:01.091549+0000","last_undegraded":"2023-09-30T02:03:01.091549+0000","last_fullsized":"2023-09-30T02:03:01.091549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:01.091362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:01.091362+0000","last_clean_scrub_stamp":"2023-09-30T02:03:01.091362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:58.825345+0000","scrub_duration":0.0030036669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:09.573555+0000","last_change":"2023-09-30T02:03:09.573487+0000","last_active":"2023-09-30T02:03:09.573555+0000","last_peered":"2023-09-30T02:03:09.573555+0000","last_clean":"2023-09-30T02:03:09.573555+0000","last_became_active":"2023-09-30T01:57:50.546760+0000","last_became_peered":"2023-09-30T01:57:50.546760+0000","last_unstale":"2023-09-30T02:03:09.573555+0000","last_undegraded":"2023-09-30T02:03:09.573555+0000","last_fullsized":"2023-09-30T02:03:09.573555+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:09.573251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:09.573251+0000","last_clean_scrub_stamp":"2023-09-30T02:03:09.573251+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:58:22.884241+0000","scrub_duration":0.0038860829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:04.715862+0000","last_change":"2023-09-30T02:03:04.715751+0000","last_active":"2023-09-30T02:03:04.715862+0000","last_peered":"2023-09-30T02:03:04.715862+0000","last_clean":"2023-09-30T02:03:04.715862+0000","last_became_active":"2023-09-30T01:57:50.549218+0000","last_became_peered":"2023-09-30T01:57:50.549218+0000","last_unstale":"2023-09-30T02:03:04.715862+0000","last_undegraded":"2023-09-30T02:03:04.715862+0000","last_fullsized":"2023-09-30T02:03:04.715862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:03:04.715486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:03:04.715486+0000","last_clean_scrub_stamp":"2023-09-30T02:03:04.715486+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:38:51.899550+0000","scrub_duration":0.0038378980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'119","reported_seq":174,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T02:03:03.560062+0000","last_change":"2023-09-30T02:03:03.559970+0000","last_active":"2023-09-30T02:03:03.560062+0000","last_peered":"2023-09-30T02:03:03.560062+0000","last_clean":"2023-09-30T02:03:03.560062+0000","last_became_active":"2023-09-30T01:57:49.478441+0000","last_became_peered":"2023-09-30T01:57:49.478441+0000","last_unstale":"2023-09-30T02:03:03.560062+0000","last_undegraded":"2023-09-30T02:03:03.560062+0000","last_fullsized":"2023-09-30T02:03:03.560062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'119","last_scrub_stamp":"2023-09-30T02:03:03.559738+0000","last_deep_scrub":"18'119","last_deep_scrub_stamp":"2023-09-30T02:03:03.559738+0000","last_clean_scrub_stamp":"2023-09-30T02:03:03.559738+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:03:56.653355+0000","scrub_duration":0.01051558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542214,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.86799999999999999,"15min":0.872},"min":{"1min":0.57199999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.93899999999999995,"15min":0.997},"min":{"1min":0.64800000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.149,"5min":1.353,"15min":1.353},"last":1.2589999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.91900000000000004,"15min":0.93700000000000006},"min":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3620000000000001,"5min":1.365,"15min":1.365},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91000000000000003,"15min":0.89500000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.97899999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97699999999999998,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.619,"15min":1.619},"last":1.3700000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.94499999999999995,"15min":0.97299999999999998},"min":{"1min":0.69099999999999995,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.79800000000000004}]},{"osd":3,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98799999999999999,"15min":1.012},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3,"5min":1.446,"15min":1.446},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.103,"15min":1.089},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.177}]},{"osd":4,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.91400000000000003,"15min":0.91800000000000004},"min":{"1min":0.63,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1779999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.054,"5min":1.083,"15min":1.091},"min":{"1min":0.76500000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3879999999999999,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.4570000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.03,"15min":1.048},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2709999999999999,"5min":1.55,"15min":1.55},"last":1.077},{"interface":"front","average":{"1min":1.044,"5min":1.077,"15min":1.093},"min":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.377,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.018}]},{"osd":7,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.157,"15min":1.1899999999999999},"min":{"1min":0.93200000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.3759999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.569},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.91400000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.601,"5min":1.627,"15min":1.627},"last":1.2310000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.91200000000000003,"15min":0.89800000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.264,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.61799999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.1830000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.92100000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.70299999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1319999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.054,"5min":1.0369999999999999,"15min":1.0609999999999999},"min":{"1min":0.86499999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.28,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.68200000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.67300000000000004,"5min":0.63,"15min":0.63},"max":{"1min":1.0489999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.006,"5min":0.94599999999999995,"15min":0.94899999999999995},"min":{"1min":0.72899999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.27,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94999999999999996,"15min":0.96599999999999997},"min":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2,"5min":1.218,"15min":1.218},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0029999999999999,"15min":1.018},"min":{"1min":0.85199999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.208,"5min":1.365,"15min":1.365},"last":1.1759999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.90300000000000002,"15min":0.94099999999999995},"min":{"1min":0.71899999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.0980000000000001},"min":{"1min":0.76300000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.258,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.1000000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.04,"15min":1.069},"min":{"1min":0.79700000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4419999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.129,"15min":1.1539999999999999},"min":{"1min":1.002,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.323,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.016}]},{"osd":6,"last update":"Sat Sep 30 02:02:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.123,"15min":1.1160000000000001},"min":{"1min":0.89500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.343,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.171,"15min":1.1859999999999999},"min":{"1min":1.0169999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.129}]}]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.85399999999999998,"15min":0.84199999999999997},"min":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94499999999999995,"15min":1.014},"min":{"1min":0.66900000000000004,"5min":0.626,"15min":0.626},"max":{"1min":1.258,"5min":1.361,"15min":1.361},"last":0.83699999999999997}]},{"osd":1,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":0.85599999999999998},"min":{"1min":0.68899999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0589999999999999,"5min":1.262,"15min":1.262},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.90500000000000003,"15min":0.877},"min":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.175,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.93600000000000005,"15min":0.93200000000000005},"min":{"1min":0.70099999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1559999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":1.0209999999999999},"min":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2869999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.181}]},{"osd":3,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.0049999999999999,"15min":0.999},"min":{"1min":0.71699999999999997,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.5389999999999999,"5min":1.575,"15min":1.575},"last":1.137},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.95499999999999996,"15min":0.95299999999999996},"min":{"1min":0.64800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.92200000000000004,"15min":0.89900000000000002},"min":{"1min":0.59799999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.222,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.0600000000000001,"15min":1.0289999999999999},"min":{"1min":0.87,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.0349999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.99199999999999999,"15min":1.0049999999999999},"min":{"1min":0.70999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.3600000000000001,"5min":1.669,"15min":1.669},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.157,"15min":1.2},"min":{"1min":0.84499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.5089999999999999,"5min":1.75,"15min":1.75},"last":0.95399999999999996}]},{"osd":7,"last update":"Sat Sep 30 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1830000000000001,"15min":1.2250000000000001},"min":{"1min":0.94599999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.4339999999999999,"5min":1.859,"15min":1.859},"last":1.083},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.036,"15min":1.032},"min":{"1min":0.78700000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.306,"5min":1.526,"15min":1.526},"last":1.1160000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":0.94099999999999995},"min":{"1min":0.67500000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1970000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.873},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.98099999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1890000000000001,"5min":1.327,"15min":1.327},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.131,"15min":1.167},"min":{"1min":0.96499999999999997,"5min":0.748,"15min":0.748},"max":{"1min":1.4410000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.238}]},{"osd":2,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.95699999999999996,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.163,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.78600000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.3500000000000001,"5min":1.472,"15min":1.472},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.997,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.4510000000000001,"5min":3.9889999999999999,"15min":3.9889999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":0.89900000000000002},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.996,"5min":1.292,"15min":1.292},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.113},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4099999999999999,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.089}]},{"osd":6,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1020000000000001,"15min":1.048},"min":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.014},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.85999999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.2549999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1319999999999999,"15min":1.129},"min":{"1min":0.96699999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.5549999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.099,"5min":1.1519999999999999,"15min":1.1679999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3100000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.042}]}]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68100000000000005,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.3049999999999999,"5min":2.29,"15min":2.29},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.753,"15min":0.80700000000000005},"min":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.89000000000000001,"5min":1.569,"15min":1.569},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70099999999999996,"15min":0.69099999999999995},"min":{"1min":0.55700000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.147,"5min":1.28,"15min":1.28},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.88200000000000001},"min":{"1min":0.66500000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.2989999999999999,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.748}]},{"osd":2,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84499999999999997,"15min":0.81999999999999995},"min":{"1min":0.63800000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.1850000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.89900000000000002,"15min":0.89600000000000002},"min":{"1min":0.79900000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.306,"5min":1.363,"15min":1.363},"last":1.0640000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.135},"min":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.391,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.069,"15min":1.083},"min":{"1min":0.75700000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.3280000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.0760000000000001},"min":{"1min":0.69399999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.3819999999999999,"5min":1.645,"15min":1.645},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.78100000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4590000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.2}]},{"osd":6,"last update":"Sat Sep 30 02:03:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.165,"15min":1.2609999999999999},"min":{"1min":0.83599999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.49,"5min":2.516,"15min":2.516},"last":1.034},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1859999999999999,"15min":1.244},"min":{"1min":0.82399999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.4399999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.88200000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:02:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1839999999999999,"15min":1.173},"min":{"1min":1.046,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.702,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.218,"15min":1.3049999999999999},"min":{"1min":0.98099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.5760000000000001,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.264}]}]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5168,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366672,"statfs":{"total":96636764160,"available":96631472128,"internally_reserved":0,"allocated":172032,"data_stored":56052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.51900000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.74299999999999999,"15min":0.77000000000000002},"min":{"1min":0.53100000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.99399999999999999,"5min":1.181,"15min":1.181},"last":0.63700000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72699999999999998,"15min":0.751},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.1459999999999999,"5min":1.284,"15min":1.284},"last":0.746},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":1.135,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.79800000000000004}]},{"osd":3,"last update":"Sat Sep 30 02:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.80400000000000005,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0680000000000001,"5min":1.452,"15min":1.452},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82199999999999995,"15min":0.85199999999999998},"min":{"1min":0.46400000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0209999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.93799999999999994}]},{"osd":4,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.98199999999999998,"15min":1.03},"min":{"1min":0.72799999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.113,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.024,"15min":1.0780000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.359,"5min":1.819,"15min":1.819},"last":0.97899999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0569999999999999,"15min":1.085},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5229999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.149},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0720000000000001,"15min":1.101},"min":{"1min":0.77000000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.4379999999999999,"5min":1.659,"15min":1.659},"last":1.0349999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.125,"15min":1.1579999999999999},"min":{"1min":0.82099999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.4650000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.129},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1439999999999999,"15min":1.1579999999999999},"min":{"1min":0.878,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":1.4259999999999999,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.214}]},{"osd":7,"last update":"Sat Sep 30 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.149,"15min":1.155},"min":{"1min":1.032,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4179999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.2130000000000001,"15min":1.226},"min":{"1min":0.85299999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.4690000000000001,"5min":1.79,"15min":1.79},"last":1.389}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5176,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366664,"statfs":{"total":96636764160,"available":96631463936,"internally_reserved":0,"allocated":180224,"data_stored":58335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91300000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.76600000000000001},"min":{"1min":0.56899999999999995,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.113,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.044}]},{"osd":2,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68300000000000005,"15min":0.72999999999999998},"min":{"1min":0.52600000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.90500000000000003,"5min":1.296,"15min":1.296},"last":0.754},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.81299999999999994,"15min":0.86599999999999999},"min":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.252,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.84099999999999997,"15min":0.874},"min":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0660000000000001,"5min":1.218,"15min":1.218},"last":0.82899999999999996}]},{"osd":4,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.97799999999999998,"15min":0.95999999999999996},"min":{"1min":0.79200000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.343,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.083,"5min":1.0820000000000001,"15min":1.1160000000000001},"min":{"1min":0.88800000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.734,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.97999999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0389999999999999,"15min":1.1200000000000001},"min":{"1min":0.76200000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.2410000000000001,"5min":1.625,"15min":1.625},"last":1.02},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.119,"15min":1.165},"min":{"1min":0.82399999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.6639999999999999,"5min":1.893,"15min":1.893},"last":1.371}]},{"osd":6,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.133,"15min":1.1930000000000001},"min":{"1min":0.84199999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.2989999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1299999999999999,"15min":1.181},"min":{"1min":0.86099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.369,"5min":1.778,"15min":1.778},"last":1.0920000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:02:57 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1850000000000001,"15min":1.232},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.6240000000000001,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.159},{"interface":"front","average":{"1min":1.151,"5min":1.1899999999999999,"15min":1.2529999999999999},"min":{"1min":0.97199999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.3300000000000001,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.137}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6140,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365700,"statfs":{"total":96636764160,"available":96630476800,"internally_reserved":0,"allocated":708608,"data_stored":585928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66700000000000004,"15min":0.64800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.871,"5min":1.006,"15min":1.006},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77200000000000002,"15min":0.755},"min":{"1min":0.45800000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.083,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88,"5min":1.087,"15min":1.087},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.81699999999999995,"15min":0.79000000000000004},"min":{"1min":0.54300000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.9729999999999999,"5min":3.9729999999999999,"15min":3.9729999999999999},"last":0.74199999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72299999999999998,"15min":0.69899999999999995},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.85199999999999998,"15min":0.874},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.99399999999999999,"15min":0.98599999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.194,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.0189999999999999,"15min":1.0429999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.288,"5min":1.448,"15min":1.448},"last":0.98799999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.98099999999999998,"15min":1.0449999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.244,"5min":1.468,"15min":1.468},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0629999999999999,"15min":1.109},"min":{"1min":0.79100000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2050000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.115}]},{"osd":6,"last update":"Sat Sep 30 02:03:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.046,"15min":1.0860000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3720000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.1080000000000001,"15min":1.143},"min":{"1min":0.92500000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.2809999999999999,"5min":1.768,"15min":1.768},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Sep 30 02:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.147,"15min":1.165},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.415,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.216,"5min":1.171,"15min":1.208},"min":{"1min":0.877,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5329999999999999,"5min":1.669,"15min":1.669},"last":1.04}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:03:22.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T02:03:22.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 Processor -- start 2023-09-30T02:03:22.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 -- start start 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f90cc06b180 0x7f90cc0ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0abe00 con 0x7f90cc06b550 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc06c8e0 con 0x7f90cc13e4e0 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cbfff700 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc06c700 con 0x7f90cc06b180 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f90cc06b550 0x7f90cc1415d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37460/0 (socket says 172.21.15.151:37460) 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 -- 172.21.15.151:0/3133307822 learned_addr learned my addr 172.21.15.151:0/3133307822 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90caffd700 1 --2- 172.21.15.151:0/3133307822 >> v2:172.21.15.172:3300/0 conn(0x7f90cc06b180 0x7f90cc0ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688371823 0 0) 0x7f90cc0abe00 con 0x7f90cc06b550 2023-09-30T02:03:22.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b8003570 con 0x7f90cc06b550 2023-09-30T02:03:22.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711931405 0 0) 0x7f90b8003570 con 0x7f90cc06b550 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 >> v2:172.21.15.172:3300/0 conn(0x7f90cc06b180 msgr2=0x7f90cc0ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 --2- 172.21.15.151:0/3133307822 >> v2:172.21.15.172:3300/0 conn(0x7f90cc06b180 0x7f90cc0ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 >> v1:172.21.15.172:6789/0 conn(0x7f90cc13e4e0 legacy=0x7f90cc13e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0a9b00 con 0x7f90cc06b550 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3603541732 0 0) 0x7f90b4001470 con 0x7f90cc06b550 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 >> v1:172.21.15.151:6789/0 conn(0x7f90cc06b550 legacy=0x7f90cc1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90b8003a60 0x7f90b8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 0x7f90b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80051c0 0x7f90b800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc06c700 con 0x7f90b8003a60 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800c0d0 con 0x7f90b80051c0 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800c290 con 0x7f90b8004810 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 0x7f90b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90caffd700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90b8003a60 0x7f90b8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cb7fe700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80051c0 0x7f90b800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 -- 172.21.15.151:0/3133307822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80051c0 msgr2=0x7f90b800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:22.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b80051c0 0x7f90b800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 -- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90b8003a60 msgr2=0x7f90b8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90b8003a60 0x7f90b8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90ca7fc700 1 -- 172.21.15.151:0/3133307822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8003570 con 0x7f90b8004810 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90cb7fe700 1 -- 172.21.15.151:0/3133307822 reap_dead start 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.610+0000 7f90caffd700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90b8003a60 0x7f90b8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90ca7fc700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 0x7f90b8004c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c390 tx=0x7f90cc0ace30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19df110920332570 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400cc20 con 0x7f90b8004810 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b400cd80 con 0x7f90b8004810 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90c9ffb700 1 -- 172.21.15.151:0/3133307822 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4005b70 con 0x7f90b8004810 2023-09-30T02:03:22.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 -- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 msgr2=0x7f90b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 0x7f90b8004c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c390 tx=0x7f90cc0ace30 comp rx=0 tx=0).stop 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 -- 172.21.15.151:0/3133307822 shutdown_connections 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3133307822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90b8004810 0x7f90b8004c80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 -- 172.21.15.151:0/3133307822 >> 172.21.15.151:0/3133307822 conn(0x7f90cc070510 msgr2=0x7f90cc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 -- 172.21.15.151:0/3133307822 shutdown_connections 2023-09-30T02:03:22.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.614+0000 7f90cbfff700 1 -- 172.21.15.151:0/3133307822 wait complete. 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 Processor -- start 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- start start 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 0x7f90cc154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc154e60 0x7f90cc13f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc1555a0 con 0x7f90cc13fe20 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc141410 con 0x7f90b8004810 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc1552b0 con 0x7f90cc154e60 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:44150/0 (socket says 172.21.15.151:44150) 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90ca7fc700 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc154e60 0x7f90cc13f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 -- 172.21.15.151:0/3349022645 learned_addr learned my addr 172.21.15.151:0/3349022645 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90caffd700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 0x7f90cc154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 -- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc154e60 msgr2=0x7f90cc13f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc154e60 0x7f90cc13f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 -- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 msgr2=0x7f90cc154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 0x7f90cc154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 -- 172.21.15.151:0/3349022645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc1447c0 con 0x7f90cc13fe20 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90caffd700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 0x7f90cc154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cb7fe700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f90cc071470 tx=0x7f90c4001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c400eea0 con 0x7f90cc13fe20 2023-09-30T02:03:22.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4006db0 con 0x7f90cc13fe20 2023-09-30T02:03:22.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c400c7f0 con 0x7f90cc13fe20 2023-09-30T02:03:22.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc140290 con 0x7f90cc13fe20 2023-09-30T02:03:22.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc1437e0 con 0x7f90cc13fe20 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.618+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc0008d0 con 0x7f90cc13fe20 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.622+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f90c400cb60 con 0x7f90cc13fe20 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.622+0000 7f90c2ffd700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 0x7f90ac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.622+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f90c407c4c0 con 0x7f90cc13fe20 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.622+0000 7f90caffd700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 0x7f90ac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:03:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.622+0000 7f90caffd700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 0x7f90ac062a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0654b0 tx=0x7f90bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:03:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.626+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f90c404cc00 con 0x7f90cc13fe20 2023-09-30T02:03:22.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.790+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f90cc0ab460 con 0x7f90cc13fe20 2023-09-30T02:03:22.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.794+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f90c404cc00 con 0x7f90cc13fe20 2023-09-30T02:03:22.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.794+0000 7f90c2ffd700 1 -- 172.21.15.151:0/3349022645 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f90c4050220 con 0x7f90cc13fe20 2023-09-30T02:03:22.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 msgr2=0x7f90ac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 0x7f90ac062a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0654b0 tx=0x7f90bc008040 comp rx=0 tx=0).stop 2023-09-30T02:03:22.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 msgr2=0x7f90cc1432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:03:22.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f90cc071470 tx=0x7f90c4001640 comp rx=0 tx=0).stop 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 shutdown_connections 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:6832/31144,v1:172.21.15.172:6833/31144] conn(0x7f90ac060550 0x7f90ac062a00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90cc13fe20 0x7f90cc1432a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90cc154e60 0x7f90cc13f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 --2- 172.21.15.151:0/3349022645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90b8004810 0x7f90cc154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 >> 172.21.15.151:0/3349022645 conn(0x7f90cc070510 msgr2=0x7f90cc13c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 shutdown_connections 2023-09-30T02:03:22.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-09-30T02:03:22.798+0000 7f90cbfff700 1 -- 172.21.15.151:0/3349022645 wait complete. 2023-09-30T02:03:22.813 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:03:22.814 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T02:03:22.814 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T02:03:22.814 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T02:03:22.815 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:22.902 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T02:03:22.902 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T02:03:22.902 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:22.988 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T02:03:22.988 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T02:03:22.988 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.071 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T02:03:23.072 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T02:03:23.072 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.158 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T02:03:23.158 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T02:03:23.158 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.236 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T02:03:23.236 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T02:03:23.236 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.308 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T02:03:23.308 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T02:03:23.308 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.391 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T02:03:23.391 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T02:03:23.391 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.467 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T02:03:23.467 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T02:03:23.467 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T02:03:23.467 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.497 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T02:03:23.498 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T02:03:23.498 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.529 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T02:03:23.529 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T02:03:23.529 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T02:03:23.530 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.548 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T02:03:23.549 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T02:03:23.549 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.608 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T02:03:23.608 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T02:03:23.608 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:03:23.628 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T02:03:23.628 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T02:03:23.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | head -n 1 2023-09-30T02:03:23.645 INFO:teuthology.orchestra.run.smithi151.stdout:1696039066.4977634 mon.a (mon.0) 96 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:03:23.646 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T02:03:23.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | head -n 1 2023-09-30T02:03:23.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | head -n 1 2023-09-30T02:03:23.723 INFO:teuthology.orchestra.run.smithi151.stdout:1696039066.4977634 mon.a (mon.0) 96 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:03:23.724 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T02:03:23.724 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T02:03:23.954 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T02:03:23.955 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T02:03:24.146 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T02:03:24.146 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T02:03:24.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T02:03:24.330 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T02:03:24.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T02:03:24.471 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T02:03:24.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T02:03:24.667 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T02:03:24.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T02:03:24.964 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T02:03:25.124 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T02:03:25.125 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T02:03:25.316 INFO:tasks.ceph:Archiving mon data... 2023-09-30T02:03:25.317 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/data/mon.a.tgz 2023-09-30T02:03:25.318 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-09-30T02:03:25.325 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.XhXzuxOK5y 2023-09-30T02:03:25.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.XhXzuxOK5y 2023-09-30T02:03:25.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.XhXzuxOK5y 2023-09-30T02:03:25.421 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.XhXzuxOK5y is 261KB 2023-09-30T02:03:25.437 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.XhXzuxOK5y 2023-09-30T02:03:25.443 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/data/mon.c.tgz 2023-09-30T02:03:25.443 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-09-30T02:03:25.492 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.XN60s1P0yE 2023-09-30T02:03:25.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.XN60s1P0yE 2023-09-30T02:03:25.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.XN60s1P0yE 2023-09-30T02:03:25.586 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.XN60s1P0yE is 261KB 2023-09-30T02:03:25.602 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.XN60s1P0yE 2023-09-30T02:03:25.610 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/data/mon.b.tgz 2023-09-30T02:03:25.610 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-09-30T02:03:25.617 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.ZOpx7kV6Gp 2023-09-30T02:03:25.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ZOpx7kV6Gp 2023-09-30T02:03:25.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.ZOpx7kV6Gp 2023-09-30T02:03:25.751 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.ZOpx7kV6Gp is 261KB 2023-09-30T02:03:25.770 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.ZOpx7kV6Gp 2023-09-30T02:03:25.776 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T02:03:25.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:03:25.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:03:25.836 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T02:03:25.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:03:25.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:03:25.869 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:03:25.873 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:03:25.880 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T02:03:25.882 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi151/crash 2023-09-30T02:03:25.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:03:25.900 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi172/crash 2023-09-30T02:03:25.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:03:25.942 INFO:tasks.ceph:Compressing logs... 2023-09-30T02:03:25.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:03:25.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:03:26.769 INFO:tasks.ceph:Archiving logs... 2023-09-30T02:03:26.770 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi151/log 2023-09-30T02:03:26.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:03:27.003 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi172/log 2023-09-30T02:03:27.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:03:27.109 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:03:27.174 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58be9e5609f64c21a1004dc3627c4052 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:03:27.177 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T02:03:27.192 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T02:03:27.298 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T02:03:27.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:03:27.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:03:27.377 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T02:03:27.386 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T02:03:27.387 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:03:27.389 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T02:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2023-09-30T02:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout:*hv01.front.sepi 44.4.53.2 2 u 1 64 377 0.087 2.284 1.246 2023-09-30T02:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout:+hv02.front.sepi 63.231.80.2 3 u 3 64 377 0.080 2.198 1.237 2023-09-30T02:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout:+hv03.front.sepi 74.6.168.72 3 u 60 64 377 0.240 0.238 0.531 2023-09-30T02:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T02:03:27.411 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T02:03:27.411 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2023-09-30T02:03:27.412 INFO:teuthology.orchestra.run.smithi172.stdout:*hv01.front.sepi 44.4.53.2 2 u 60 64 377 0.072 -0.288 0.193 2023-09-30T02:03:27.412 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 64 377 0.060 -0.115 0.329 2023-09-30T02:03:27.412 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 377 0.082 -0.601 0.377 2023-09-30T02:03:27.412 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T02:03:27.414 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T02:03:27.422 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T02:03:27.422 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T02:03:27.431 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T02:03:27.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T02:03:27.449 INFO:teuthology.task.internal:Duration was 881.821700 seconds 2023-09-30T02:03:27.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T02:03:27.499 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T02:03:27.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:03:27.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:03:27.553 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T02:03:27.554 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2023-09-30T02:03:27.554 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:03:27.567 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-09-30T02:03:27.567 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:03:27.581 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T02:03:27.581 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:03:27.584 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:03:27.660 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T02:03:27.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:03:27.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:03:27.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T02:03:27.739 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T02:03:27.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:03:27.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:03:27.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T02:03:27.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:03:27.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:03:27.832 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2023-09-30T02:03:27.833 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-09-30T02:03:27.861 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:03:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:03:27.867 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:03:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:03:27.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T02:03:27.891 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T02:03:27.892 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi151 2023-09-30T02:03:27.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:03:27.916 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135/remote/smithi172 2023-09-30T02:03:27.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:03:27.937 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T02:03:27.938 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:03:27.956 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:03:27.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T02:03:27.973 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T02:03:27.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T02:03:28.020 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T02:03:28.020 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:03:28.023 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:03:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:03 /home/ubuntu/cephtest 2023-09-30T02:03:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:03 /home/ubuntu/cephtest 2023-09-30T02:03:28.033 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T02:03:28.062 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T02:03:28.080 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T02:03:28.107 DEBUG:teuthology.nuke:shortname: smithi151 2023-09-30T02:03:28.108 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:03:28.128 DEBUG:teuthology.nuke:shortname: smithi172 2023-09-30T02:03:28.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:03:28.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:38:57.558491', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T02:03:28.148 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407135', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:38:57.557219', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T02:03:28.185 INFO:teuthology.orchestra.console.smithi151:Power off 2023-09-30T02:03:28.186 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:03:28.200 INFO:teuthology.orchestra.console.smithi172:Power off 2023-09-30T02:03:28.201 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:03:28.209 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2023-09-30T02:03:28.209 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:28.222 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-09-30T02:03:28.223 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:28.232 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-09-30T02:03:28.245 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-09-30T02:03:32.233 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:32.246 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:32.344 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-09-30T02:03:32.356 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-09-30T02:03:36.346 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:36.357 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:36.455 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-09-30T02:03:36.466 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-09-30T02:03:36.567 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-09-30T02:03:40.457 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:03:40.566 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2023-09-30T02:03:40.667 INFO:teuthology.orchestra.console.smithi151:Power off completed 2023-09-30T02:03:40.771 INFO:teuthology.run:Summary data: description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} duration: 881.8216996192932 failure_reason: '"1696039066.4977634 mon.a (mon.0) 96 : cluster [ERR] Health check failed: Module ''pg_autoscaler'' has failed: ''PgAutoscaler'' object has no attribute ''noautoscale'' (MGR_MODULE_ERROR)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=58be9e5609f64c21a1004dc3627c4052 status: fail success: false 2023-09-30T02:03:40.772 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T02:03:40.848 INFO:teuthology.run:FAIL